Original: A UNIFYING VIEW ON IMPLICIT BIAS IN TRAINING LINEAR NEURAL NETWORKS
Revised: A UNIFYING VIEW ON IMPLICIT BIAS IN TRAINING LINEAR NEURAL NETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the implicit bias of gradient ﬂow (i e., gradient descent with inﬁnitesimal step size) on linear neural network training.
Revised: We study the implicit bias of gradient ﬂow (i e., gradient descent with inﬁnitesimal step size) on linear neural network training.

Original: We propose a tensor formulation of neural networks that includes fully-connected, diagonal, and convolutional networks as special cases, and investigate the linear version of the formulation called linear tensor networks.
Revised: We propose a tensor formulation of neural networks that includes fully-connected, diagonal, and convolutional networks as special cases, and investigate the linear version of the formulation called linear tensor networks.

Original: With this formulation, we can characterize the convergence direction of the network parameters as singular vectors of a tensor deﬁned by the network.
Revised: With this formulation, we can characterize the convergence direction of the network parameters as singular vectors of a tensor deﬁned by the network.

Original: For L -layer linear tensor networks that are orthogonally decomposable, we show that gradient ﬂow on separable classiﬁcation ﬁnds a stationary point of the (cid:96) 2 /L max-margin problem in a “transformed” input space deﬁned by the network.
Revised: For L -layer linear tensor networks that are orthogonally decomposable, we show that gradient ﬂow on separable classiﬁcation ﬁnds a stationary point of the (cid:96) 2 /L max-margin problem in a “transformed” input space deﬁned by the network.

Original: For underdetermined regression, we prove that gradient ﬂow ﬁnds a global minimum which minimizes a norm-like function that interpolates between weighted (cid:96) 1 and (cid:96) 2 norms in the transformed input space.
Revised: For underdetermined regression, we prove that gradient ﬂow ﬁnds a global minimum which minimizes a norm-like function that interpolates between weighted (cid:96) 1 and (cid:96) 2 norms in the transformed input space.

Original: Our theorems subsume existing results in the literature while removing standard convergence assumptions.
Revised: Our theorems subsume existing results in the literature while removing standard convergence assumptions.

Original: We also provide experiments that corroborate our analysis.
Revised: We also provide experiments that corroborate our analysis.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Overparametrized neural networks have inﬁnitely many solutions that achieve zero training error, and such global minima have different generalization performance.
Revised: Overparametrized neural networks have inﬁnitely many solutions that achieve zero training error, and such global minima have different generalization performance.

Original: Moreover, training a neural network is a high-dimensional nonconvex problem, which is typically intractable to solve.
Revised: Moreover, training a neural network is a high-dimensional nonconvex problem, which is typically intractable to solve.

Original: However, the success of deep learning indicates that ﬁrst-order methods such as gradient descent or stochastic gradient descent (GD/SGD) not only (a) succeed in ﬁnding global minima, but also (b) are biased towards solutions that generalize well, which largely has remained a mystery in the literature.
Revised: However, the success of deep learning indicates that ﬁrst-order methods such as gradient descent or stochastic gradient descent (GD/SGD) not only (a) succeed in ﬁnding global minima, but also (b) are biased towards solutions that generalize well, which largely has remained a mystery in the literature.

Original: To explain part (a) of the phenomenon, there is a growing literature studying the convergence of
Revised: To explain part (a) of the phenomenon, there is a growing literature studying the convergence of

Original: GD/SGD on overparametrized neural networks (e.g., Du et al.
Revised: GD/SGD on overparametrized neural networks (e.g., Du et al.

Original: (2018a;b); Allen-Zhu et al.
Revised: (2018a;b); Allen-Zhu et al.

Original: Zou et al.
Revised: Zou et al.

Original: ; Jacot et al.
Revised: ; Jacot et al.

Original: ; Oymak & Soltanolkotabi (2020), and many more).
Revised: ; Oymak & Soltanolkotabi (2020), and many more).

Original: There are also convergence results that focus on linear networks, without nonlinear activations (Bartlett et al., 2018; Arora et al., 2019a;
Revised: There are also convergence results that focus on linear networks, without nonlinear activations (Bartlett et al., 2018; Arora et al., 2019a;

Original: Wu et al., 2019; Du & Hu, 2019;
Revised: Wu et al., 2019; Du & Hu, 2019;

Original: Hu et al., 2020).
Revised: Hu et al., 2020).

Original: These results typically focus on the convergence of loss, hence do not address which of the many global minima is reached.
Revised: These results typically focus on the convergence of loss, hence do not address which of the many global minima is reached.

Original: Another line of results tackles part (b), by studying the implicit bias or regularization of gradientbased methods on neural networks or related problems (Gunasekar et al., 2017; 2018a;b; Arora et al., 2018; Soudry et al., 2018; Ji & Telgarsky, 2019a; Arora et al., 2019b; Woodworth et al., 2020; Chizat & Bach, 2020;
Revised: Another line of results tackles part (b), by studying the implicit bias or regularization of gradientbased methods on neural networks or related problems (Gunasekar et al., 2017; 2018a;b; Arora et al., 2018; Soudry et al., 2018; Ji & Telgarsky, 2019a; Arora et al., 2019b; Woodworth et al., 2020; Chizat & Bach, 2020;

Original: Gissin et al., 2020).
Revised: Gissin et al., 2020).

Original: These results have shown interesting progress that even without explicit regularization terms in the training objective, algorithms such as GD applied on neural networks have an implicit bias towards certain solutions among the many global minima.
Revised: These results have shown interesting progress that even without explicit regularization terms in the training objective, algorithms such as GD applied on neural networks have an implicit bias towards certain solutions among the many global minima.

Original: However, in proving such results, many results rely on convergence assumptions such as global convergence of loss to zero and/or directional convergence of parameters and gradients, which ideally should be removed.
Revised: However, in proving such results, many results rely on convergence assumptions such as global convergence of loss to zero and/or directional convergence of parameters and gradients, which ideally should be removed.

Original: We cover more details on this literature in Section 2.2.
Revised: We cover more details on this literature in Section 2.2.

Original: 1. S UMMARY OF OUR CONTRIBUTIONS
Revised: 1. S UMMARY OF OUR CONTRIBUTIONS

Original: We study the implicit bias of gradient ﬂow (GD with inﬁnitesimal step size) on linear neural networks.
Revised: We study the implicit bias of gradient ﬂow (GD with inﬁnitesimal step size) on linear neural networks.

Original: Following recent progress on this topic, we consider classiﬁcation and regression problems that have multiple solutions with zero training error.
Revised: Following recent progress on this topic, we consider classiﬁcation and regression problems that have multiple solutions with zero training error.

Original: Our analyses apply to a general class of networks, and prove both convergence and bias, overcoming the limitations of the existing results .
Revised: Our analyses apply to a general class of networks, and prove both convergence and bias, overcoming the limitations of the existing results .

Original: • We propose a general tensor formulation of nonlinear neural networks which includes many network architectures considered in the literature.
Revised: • We propose a general tensor formulation of nonlinear neural networks which includes many network architectures considered in the literature.

Original: In this paper, we focus on the linear version of this formulation (i.e., no nonlinear activations), called linear tensor networks .
Revised: In this paper, we focus on the linear version of this formulation (i.e., no nonlinear activations), called linear tensor networks .

Original: • For linearly separable classiﬁcation, we prove that linear tensor network parameters converge in direction to singular vectors of a tensor deﬁned by the network.
Revised: • For linearly separable classiﬁcation, we prove that linear tensor network parameters converge in direction to singular vectors of a tensor deﬁned by the network.

Original: As a corollary , we show that linear fully-connected networks converge to the (cid:96) 2 max-margin solution (Ji & Telgarsky, 2020).
Revised: As a corollary , we show that linear fully-connected networks converge to the (cid:96) 2 max-margin solution (Ji & Telgarsky, 2020).

Original: • For separable classiﬁcation, we further show that if the linear tensor network is orthogonally decomposable (Assumption 1), the gradient ﬂow ﬁnds the (cid:96) 2 / depth max-margin solution in the singular value space, leading the parameters to converge to the top singular vectors of the tensor when depth = 2 .
Revised: • For separable classiﬁcation, we further show that if the linear tensor network is orthogonally decomposable (Assumption 1), the gradient ﬂow ﬁnds the (cid:96) 2 / depth max-margin solution in the singular value space, leading the parameters to converge to the top singular vectors of the tensor when depth = 2 .

Original: This theorem subsumes known results on linear convolutional networks and diagonal networks proved in Gunasekar et al.
Revised: This theorem subsumes known results on linear convolutional networks and diagonal networks proved in Gunasekar et al.

Original: (2018b), without using convergence assumptions.
Revised: (2018b), without using convergence assumptions.

Original: • For underdetermined linear regression, we study the limit points of gradient ﬂow on orthogonally decomposable networks (Assumption 1), and provide a full characterization of the limit points.
Revised: • For underdetermined linear regression, we study the limit points of gradient ﬂow on orthogonally decomposable networks (Assumption 1), and provide a full characterization of the limit points.

Original: This theorem covers results on deep matrix sensing (Arora et al., 2019b) as a special case, and extends a similar recent result (Woodworth et al., 2020) to a broader class of networks.
Revised: This theorem covers results on deep matrix sensing (Arora et al., 2019b) as a special case, and extends a similar recent result (Woodworth et al., 2020) to a broader class of networks.

Original: • For underdetermined linear regression with deep linear fully-connected networks, we prove that the network converges to the minimum (cid:96) 2 norm solutions as we scale the initialization to zero.
Revised: • For underdetermined linear regression with deep linear fully-connected networks, we prove that the network converges to the minimum (cid:96) 2 norm solutions as we scale the initialization to zero.

Original: • Lastly, we present simple experiments that corroborate our theoretical analysis.
Revised: • Lastly, we present simple experiments that corroborate our theoretical analysis.

Original: Figure 1 shows that our predictions of limit points match with solutions found by GD.
Revised: Figure 1 shows that our predictions of limit points match with solutions found by GD.

Original:  P ROBLEM SETTINGS AND RELATED WORKS
Revised:  P ROBLEM SETTINGS AND RELATED WORKS

Original: We ﬁrst deﬁne notation used in the paper.
Revised: We ﬁrst deﬁne notation used in the paper.

Original: Given a positive integer a , let [ a ] := { 1 , . . .
Revised: Given a positive integer a , let [ a ] := { 1 , . . .

Original: , a } .
Revised: , a } .

Original: We use
Revised: We use

Original: I d to denote the d × d identity matrix.
Revised: I d to denote the d × d identity matrix.

Original: Given a matrix A , we use vec( A ) to denote its vectorization, i.e., the concatenation of all columns of A .
Revised: Given a matrix A , we use vec( A ) to denote its vectorization, i.e., the concatenation of all columns of A .

Original: For two vectors a and b , let a ⊗ b be their tensor product, a (cid:12) b be their element-wise product, and a (cid:12) k be the element-wise k -th power of a .
Revised: For two vectors a and b , let a ⊗ b be their tensor product, a (cid:12) b be their element-wise product, and a (cid:12) k be the element-wise k -th power of a .

Original: Given an order- L tensor A ∈ R k 1 ×···× k L , we use [ A ] j 1 ,...,j L to denote ( j 1 , j 2 , . . .
Revised: Given an order- L tensor A ∈ R k 1 ×···× k L , we use [ A ] j 1 ,...,j L to denote ( j 1 , j 2 , . . .

Original: , j L ) -th element of A , where j l ∈ [ k l ] for all l ∈
Revised: , j L ) -th element of A , where j l ∈ [ k l ] for all l ∈

Original: [ L ] .
Revised: [ L ] .

Original: In element indexing, we use · to denote all indices in the corresponding dimension, and a : b to denote all indices from a to b .
Revised: In element indexing, we use · to denote all indices in the corresponding dimension, and a : b to denote all indices from a to b .

Original: For example, for a matrix A , [ A ] · , 4:6 denotes a submatrix that consists of 4th–6th columns of A .
Revised: For example, for a matrix A , [ A ] · , 4:6 denotes a submatrix that consists of 4th–6th columns of A .

Original: The square bracket notation for indexing overloads with [ a ] when a ∈ N , but they will be distinguishable from the context.
Revised: The square bracket notation for indexing overloads with [ a ] when a ∈ N , but they will be distinguishable from the context.

Original: Since element indices start from 1 , we re-deﬁne the modulo operation a mod d := a − (cid:98) a − 1 d (cid:99) d ∈ [ d ] for a > 0 .
Revised: Since element indices start from 1 , we re-deﬁne the modulo operation a mod d := a − (cid:98) a − 1 d (cid:99) d ∈ [ d ] for a > 0 .

Original: We use e kj to denote the j -th stardard basis vector of the vector space R k .
Revised: We use e kj to denote the j -th stardard basis vector of the vector space R k .

Original: Given a tensor A ∈ R k 1 ×···× k L and linear maps B l ∈ R p
Revised: Given a tensor A ∈ R k 1 ×···× k L and linear maps B l ∈ R p

Original: l × k l for l ∈ [ L ] , we deﬁne the multilinear multiplication ◦ between them as
Revised: l × k l for l ∈ [ L ] , we deﬁne the multilinear multiplication ◦ between them as

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: j 1 ,...,j L
Revised: j 1 ,...,j L

Original: [ A ] j 1 ,...,j L ( e k 1 j 1 ⊗ · · · ⊗ e k L j L ) ◦ ( B T 1 , . . .
Revised: [ A ] j 1 ,...,j L ( e k 1 j 1 ⊗ · · · ⊗ e k L j L ) ◦ ( B T 1 , . . .

Original: , B TL ) :
Revised: , B TL ) :

Original: = (cid:88)
Revised: = (cid:88)

Original: j 1 ,...,j L
Revised: j 1 ,...,j L

Original: [ A ] j 1 ,...,j L ( B 1 e k 1 j 1 ⊗ · · · ⊗
Revised: [ A ] j 1 ,...,j L ( B 1 e k 1 j 1 ⊗ · · · ⊗

Original: B L e k L j L ) ∈
Revised: B L e k L j L ) ∈

Original: R p 1 ×···× p L .
Revised: R p 1 ×···× p L .

Original: 2. P ROBLEM SETTINGS
Revised: 2. P ROBLEM SETTINGS

Original: We are given a dataset { ( x i , y i ) } ni =1 , where x i ∈ R d and y
Revised: We are given a dataset { ( x i , y i ) } ni =1 , where x i ∈ R d and y

Original: i ∈ R .
Revised: i ∈ R .

Original: We let X ∈ R n × d and y ∈ R n be the data matrix and label vector, respectively.
Revised: We let X ∈ R n × d and y ∈ R n be the data matrix and label vector, respectively.

Original: We study binary classiﬁcation and linear regression in this paper, in order to focus on the settings where there exist many global solutions.
Revised: We study binary classiﬁcation and linear regression in this paper, in order to focus on the settings where there exist many global solutions.

Original: For binary classiﬁcation, we assume y i ∈ {± 1 } and that the data is separable: there exists a unit vector z
Revised: For binary classiﬁcation, we assume y i ∈ {± 1 } and that the data is separable: there exists a unit vector z

Original: such that y i
Revised: such that y i

Original: x
Revised: x

Original: Ti z ≥ γ > 0 for all i ∈ [ n ] .
Revised: Ti z ≥ γ > 0 for all i ∈ [ n ] .

Original: For regression, we consider the underdetermined case ( n ≤ d ) where there are many parameters z ∈ R d such that Xz =
Revised: For regression, we consider the underdetermined case ( n ≤ d ) where there are many parameters z ∈ R d such that Xz =

Original: y .
Revised: y .

Original: We assume X has full row rank.
Revised: We assume X has full row rank.

Original: We use f ( · ; Θ ) : R d → R to denote a model parametrized by Θ .
Revised: We use f ( · ; Θ ) : R d → R to denote a model parametrized by Θ .

Original: Given the network and the dataset, we consider minimizing the training loss L ( Θ ) :
Revised: Given the network and the dataset, we consider minimizing the training loss L ( Θ ) :

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =1 (cid:96) ( f ( x i ; Θ ) , y i ) over Θ .
Revised: =1 (cid:96) ( f ( x i ; Θ ) , y i ) over Θ .

Original: Following previous results (e.g., Lyu & Li (2020); Ji & Telgarsky (2020)), we use the exponential loss (cid:96) (ˆ y, y ) =
Revised: Following previous results (e.g., Lyu & Li (2020); Ji & Telgarsky (2020)), we use the exponential loss (cid:96) (ˆ y, y ) =

Original: exp( − ˆ yy ) for classiﬁcation problems.
Revised: exp( − ˆ yy ) for classiﬁcation problems.

Original: For regression, we use the squared error loss (cid:96) (ˆ y, y )
Revised: For regression, we use the squared error loss (cid:96) (ˆ y, y )

Original: =
Revised: =

Original: 12 (ˆ y − y ) 2 .
Revised: 12 (ˆ y − y ) 2 .

Original: On the algorithm side, we minimize L using gradient ﬂow, which can be viewed as GD with inﬁnitesimal step size.
Revised: On the algorithm side, we minimize L using gradient ﬂow, which can be viewed as GD with inﬁnitesimal step size.

Original: The gradient ﬂow dynamics is deﬁned as ddt Θ
Revised: The gradient ﬂow dynamics is deﬁned as ddt Θ

Original: = −∇ Θ L ( Θ ) .
Revised: = −∇ Θ L ( Θ ) .

Original: 2. R ELATED WORKS
Revised: 2. R ELATED WORKS

Original: Gradient ﬂow/descent in separable classiﬁcation.
Revised: Gradient ﬂow/descent in separable classiﬁcation.

Original: For linear models f ( x ; z ) =
Revised: For linear models f ( x ; z ) =

Original: x T z with separable data, Soudry et al.
Revised: x T z with separable data, Soudry et al.

Original: (2018) show that the GD run on L drives (cid:107) z (cid:107) to ∞ , but z converges in direction to the (cid:96) 2 max-margin classiﬁer.
Revised: (2018) show that the GD run on L drives (cid:107) z (cid:107) to ∞ , but z converges in direction to the (cid:96) 2 max-margin classiﬁer.

Original: The limit direction of z is aligned with the solution of minimize z ∈ R d (cid:107) z (cid:107) subject to y i x
Revised: The limit direction of z is aligned with the solution of minimize z ∈ R d (cid:107) z (cid:107) subject to y i x

Original: Ti z ≥ 1 for i ∈ [ n ] , (1) where the norm in the cost is the (cid:96) 2 norm.
Revised: Ti z ≥ 1 for i ∈ [ n ] , (1) where the norm in the cost is the (cid:96) 2 norm.

Original: Nacson et al.
Revised: Nacson et al.

Original: (2019b;c); Gunasekar et al.
Revised: (2019b;c); Gunasekar et al.

Original: (2018a)
Revised: (2018a)

Original: Ji &
Revised: Ji &

Original: Telgarsky (2019b;c) extend these results to other (stochastic) algorithms and non-separable settings.
Revised: Telgarsky (2019b;c) extend these results to other (stochastic) algorithms and non-separable settings.

Original: Gunasekar et al.
Revised: Gunasekar et al.

Original: (2018b) study the same problem on linear neural networks and show that GD exhibits different implicit bias depending on the architecture.
Revised: (2018b) study the same problem on linear neural networks and show that GD exhibits different implicit bias depending on the architecture.

Original: The authors show that the linear coefﬁcients of the network converges in direction to the solution of (1) with different norms: (cid:96) 2 norm for linear fully-connected networks, (cid:96) 2 /L
Revised: The authors show that the linear coefﬁcients of the network converges in direction to the solution of (1) with different norms: (cid:96) 2 norm for linear fully-connected networks, (cid:96) 2 /L

Original: (quasi-)norm for diagonal networks, and DFT-domain (cid:96) 2 /L
Revised: (quasi-)norm for diagonal networks, and DFT-domain (cid:96) 2 /L

Original: (quasi-)norm for convolutional networks with full-length ﬁlters.
Revised: (quasi-)norm for convolutional networks with full-length ﬁlters.

Original: Here, L denotes the depth.
Revised: Here, L denotes the depth.

Original: We note that Gunasekar et al.
Revised: We note that Gunasekar et al.

Original: (2018b) assume that GD globally minimizes the loss, and the network parameters and the gradient with respect to the linear coefﬁcients converge in direction.
Revised: (2018b) assume that GD globally minimizes the loss, and the network parameters and the gradient with respect to the linear coefﬁcients converge in direction.

Original: Subsequent results (Ji & Telgarsky, 2019a; 2020) remove such assumptions for linear fully-connected networks.
Revised: Subsequent results (Ji & Telgarsky, 2019a; 2020) remove such assumptions for linear fully-connected networks.

Original: A recent line of results (Nacson et al., 2019a; Lyu & Li, 2020; Ji & Telgarsky, 2020) study general homogeneous models and show divergence of parameters to inﬁnity, monotone increase of smoothed margin, directional convergence and alignment of parameters (see Section 4 for details).
Revised: A recent line of results (Nacson et al., 2019a; Lyu & Li, 2020; Ji & Telgarsky, 2020) study general homogeneous models and show divergence of parameters to inﬁnity, monotone increase of smoothed margin, directional convergence and alignment of parameters (see Section 4 for details).

Original: Lyu & Li (2020) also characterize the limit direction of parameters as the KKT point of a max-margin problem similar to (1), but this characterization does not provide useful insights for the functions f ( · ; Θ ) represented by speciﬁc architectures, because the formulation is in the parameter space Θ .
Revised: Lyu & Li (2020) also characterize the limit direction of parameters as the KKT point of a max-margin problem similar to (1), but this characterization does not provide useful insights for the functions f ( · ; Θ ) represented by speciﬁc architectures, because the formulation is in the parameter space Θ .

Original: Also, these results require that gradient ﬂow/descent has already reached 100% training accuracy.
Revised: Also, these results require that gradient ﬂow/descent has already reached 100% training accuracy.

Original: Although we study a more restrictive set of networks (i.e., deep linear), we provide a more complete characterization of the implicit bias for the functions f ( · ; Θ ) , without assuming 100% training accuracy.
Revised: Although we study a more restrictive set of networks (i.e., deep linear), we provide a more complete characterization of the implicit bias for the functions f ( · ; Θ ) , without assuming 100% training accuracy.

Original: Gradient ﬂow/descent in linear regression.
Revised: Gradient ﬂow/descent in linear regression.

Original: It is known that for linear models f ( x ; z ) =
Revised: It is known that for linear models f ( x ; z ) =

Original: x T z ,
Revised: x T z ,

Original: GD converges to the global minimum that is closest in (cid:96) 2 distance to the initialization (see e.g.,
Revised: GD converges to the global minimum that is closest in (cid:96) 2 distance to the initialization (see e.g.,

Original: Gunasekar et al.
Revised: Gunasekar et al.

Original: (2018a)).
Revised: (2018a)).

Original: However, relatively less is known for deep networks, even for linear networks.
Revised: However, relatively less is known for deep networks, even for linear networks.

Original: This is partly because the parameters do not diverge to inﬁnity, hence making limit points highly dependent on the initialization; this dependency renders analysis difﬁcult.
Revised: This is partly because the parameters do not diverge to inﬁnity, hence making limit points highly dependent on the initialization; this dependency renders analysis difﬁcult.

Original: A related problem of matrix sensing aims to minimize (cid:80) ni =

Revised: A related problem of matrix sensing aims to minimize (cid:80) ni =

Original: ( y i − (cid:104)
Revised: ( y i − (cid:104)

Original: A i , W 1 · · · W L (cid:105) ) 2 over W 1 , . . .
Revised: A i , W 1 · · · W L (cid:105) ) 2 over W 1 , . . .

Original: , W L ∈ R d × d .
Revised: , W L ∈ R d × d .

Original: It is shown in Gunasekar et al.
Revised: It is shown in Gunasekar et al.

Original: ; Arora et al. (2019b) that if the sensor matrices A i commute and we initialize W l to α I , GD ﬁnds the minimum nuclear norm solution as α → 0 .
Revised: ; Arora et al. (2019b) that if the sensor matrices A i commute and we initialize W l to α I , GD ﬁnds the minimum nuclear norm solution as α → 0 .

Original: Chizat et al.
Revised: Chizat et al.

Original: (2019) shows that if a network is zero at initialization, and we scale the network output by a factor of α → ∞ , then the GD dynamics enters a “lazy regime” where the network behaves like a ﬁrst-order approximation at its initialization, as also seen in results studying kernel approximations of neural networks and convergence of GD in the corresponding RKHS (e.g., Jacot et al.
Revised: (2019) shows that if a network is zero at initialization, and we scale the network output by a factor of α → ∞ , then the GD dynamics enters a “lazy regime” where the network behaves like a ﬁrst-order approximation at its initialization, as also seen in results studying kernel approximations of neural networks and convergence of GD in the corresponding RKHS (e.g., Jacot et al.

Original: Woodworth et al.
Revised: Woodworth et al.

Original: (2020) study linear regression with a diagonal network of the form f ( x ; w + , w − ) =
Revised: (2020) study linear regression with a diagonal network of the form f ( x ; w + , w − ) =

Original: x T ( w (cid:12) L + − w (cid:12) L − ) , where w + and w − are identically initialized w + (0)
Revised: x T ( w (cid:12) L + − w (cid:12) L − ) , where w + and w − are identically initialized w + (0)

Original: =
Revised: =

Original: w − (0) =
Revised: w − (0) =

Original: α ¯ w .
Revised: α ¯ w .

Original: The authors show that the global minimum reached by GD minimizes a normlike function which interpolates between (weighted) (cid:96) 1 norm ( α → 0 ) and (cid:96) 2 norm ( α → ∞ ).
Revised: The authors show that the global minimum reached by GD minimizes a normlike function which interpolates between (weighted) (cid:96) 1 norm ( α → 0 ) and (cid:96) 2 norm ( α → ∞ ).

Original: In our paper, we consider a more general class of orthogonally decomposable networks, and obtain similar results interpolating between weighted (cid:96) 1 and (cid:96) 2 norms.
Revised: In our paper, we consider a more general class of orthogonally decomposable networks, and obtain similar results interpolating between weighted (cid:96) 1 and (cid:96) 2 norms.

Original: We also remark that our results include the results in Arora et al.
Revised: We also remark that our results include the results in Arora et al.

Original: (2019b)
Revised: (2019b)

Original: as a special case, and we do not assume convergence to global minima, as done in Gunasekar et al.
Revised: as a special case, and we do not assume convergence to global minima, as done in Gunasekar et al.

Original: ; Arora et al. (2019b); Woodworth et al.
Revised: ; Arora et al. (2019b); Woodworth et al.

Original:  T ENSOR FORMULATION OF NEURAL NETWORKS
Revised:  T ENSOR FORMULATION OF NEURAL NETWORKS

Original: In this section, we present a general tensor formulation of neural networks.
Revised: In this section, we present a general tensor formulation of neural networks.

Original: Given an input x ∈ R d , the network uses a linear map M that maps x to a order- L tensor M ( x ) ∈ R k 1 ×···× k L , where L ≥ 2 .
Revised: Given an input x ∈ R d , the network uses a linear map M that maps x to a order- L tensor M ( x ) ∈ R k 1 ×···× k L , where L ≥ 2 .

Original: Using parameters v l ∈ R k l and activation φ , the network computes its layers as the following:
Revised: Using parameters v l ∈ R k l and activation φ , the network computes its layers as the following:

Original: [Equation]
Revised: [Equation]

Original: We use Θ to denote the collection of all parameters ( v 1 , . .
Revised: We use Θ to denote the collection of all parameters ( v 1 , . .

Original: . , v L ) .
Revised: . , v L ) .

Original: We call M ( x ) the data tensor .
Revised: We call M ( x ) the data tensor .

Original: Although this new formulation may look a bit odd in the ﬁrst glance, it is general enough to capture many network architectures considered in the literature, including fully-connected networks, diagonal networks, and circular convolutional networks.
Revised: Although this new formulation may look a bit odd in the ﬁrst glance, it is general enough to capture many network architectures considered in the literature, including fully-connected networks, diagonal networks, and circular convolutional networks.

Original: We formally deﬁne these architectures below.
Revised: We formally deﬁne these architectures below.

Original: Diagonal networks.
Revised: Diagonal networks.

Original: An L -layer diagonal network is written as
Revised: An L -layer diagonal network is written as

Original: [Equation]
Revised: [Equation]

Original: where w l ∈ R d for l ∈
Revised: where w l ∈ R d for l ∈

Original: [ L ] .
Revised: [ L ] .

Original: The representation of f diag as the tensor form (2) is straightforward.
Revised: The representation of f diag as the tensor form (2) is straightforward.

Original: Let
Revised: Let

Original: M diag ( x ) ∈ R d ×···× d have [ M diag ( x )] j,j,...,j
Revised: M diag ( x ) ∈ R d ×···× d have [ M diag ( x )] j,j,...,j

Original: =
Revised: =

Original: [ x ] j , while all the remaining entries of M diag ( x ) are set to zero.
Revised: [ x ] j , while all the remaining entries of M diag ( x ) are set to zero.

Original: We can set v l = w l for all l , and M = M diag to verify that (2) and (3) are equivalent.
Revised: We can set v l = w l for all l , and M = M diag to verify that (2) and (3) are equivalent.

Original: Circular convolutional networks.
Revised: Circular convolutional networks.

Original: The tensor formulation (2) includes convolutional networks
Revised: The tensor formulation (2) includes convolutional networks

Original: [Equation]
Revised: [Equation]

Original: where w l ∈ R k l with k l ≤ d and k L = d , and (cid:63) deﬁnes the circular convolution: for any a ∈ R d and b ∈ R k ( k ≤ d ), we have a (cid:63) b ∈ R d deﬁned as [ a (cid:63) b ] i = (cid:80) kj =

Revised: where w l ∈ R k l with k l ≤ d and k L = d , and (cid:63) deﬁnes the circular convolution: for any a ∈ R d and b ∈ R k ( k ≤ d ), we have a (cid:63) b ∈ R d deﬁned as [ a (cid:63) b ] i = (cid:80) kj =

Original: [ a ] ( i + j − 1) mod d [ b ] j , for i ∈ [ d ] .
Revised: [ a ] ( i + j − 1) mod d [ b ] j , for i ∈ [ d ] .

Original: Deﬁne M conv ( x ) ∈ R k 1 ×···× k L as [ M conv ( x )] j 1 ,j 2 ,...,j L =
Revised: Deﬁne M conv ( x ) ∈ R k 1 ×···× k L as [ M conv ( x )] j 1 ,j 2 ,...,j L =

Original: [ x ] ( (cid:80) Ll =
Revised: [ x ] ( (cid:80) Ll =

Original: j l − L +1)
Revised: j l − L +1)

Original: mod d for j l ∈ [ k l ] , l ∈
Revised: mod d for j l ∈ [ k l ] , l ∈

Original: [ L ] .
Revised: [ L ] .

Original: Setting v l = w l and M = M conv , we can verify that (2) and (4) are identical.
Revised: Setting v l = w l and M = M conv , we can verify that (2) and (4) are identical.

Original: Fully-connected networks.
Revised: Fully-connected networks.

Original: An L -layer fully-connected network is deﬁned as
Revised: An L -layer fully-connected network is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where W l ∈ R d l × d l +1 for l ∈
Revised: where W l ∈ R d l × d l +1 for l ∈

Original: [ L − 1] (we use d 1 = d ) and w L ∈ R d L .
Revised: [ L − 1] (we use d 1 = d ) and w L ∈ R d L .

Original: One can represent f fc as the tensor form (2) by deﬁning parameters v l = vec( W l ) for l ∈
Revised: One can represent f fc as the tensor form (2) by deﬁning parameters v l = vec( W l ) for l ∈

Original: [ L − 1] and v L = w L , and constructing the tensor M fc ( x ) by a recursive “block diagonal” manner.
Revised: [ L − 1] and v L = w L , and constructing the tensor M fc ( x ) by a recursive “block diagonal” manner.

Original: For example, if L = 2 , we can deﬁne M fc ( x ) ∈ R d 1 d 2 × d 2 to be the Kronecker product of I d 2 and x . For deeper networks, we defer the full description of M fc ( x ) to Appendix B.
Revised: For example, if L = 2 , we can deﬁne M fc ( x ) ∈ R d 1 d 2 × d 2 to be the Kronecker product of I d 2 and x . For deeper networks, we defer the full description of M fc ( x ) to Appendix B.

Original: Our focus: linear tensor networks.
Revised: Our focus: linear tensor networks.

Original: Throughout this section, we have used the activation φ to motivate our tensor formulation (2) for neural networks with nonlinear activations.
Revised: Throughout this section, we have used the activation φ to motivate our tensor formulation (2) for neural networks with nonlinear activations.

Original: For the remaining of the paper, we study the case whose activation is linear , i.e., φ ( t ) =
Revised: For the remaining of the paper, we study the case whose activation is linear , i.e., φ ( t ) =

Original: t .
Revised: t .

Original: In this case,
Revised: In this case,

Original: [Equation]
Revised: [Equation]

Original: We will refer to (6) as linear tensor networks , where “linear” is to indicate that the activation is linear.
Revised: We will refer to (6) as linear tensor networks , where “linear” is to indicate that the activation is linear.

Original: Note that as a function of parameters v 1 , . .
Revised: Note that as a function of parameters v 1 , . .

Original: . , v L , f ( x ; Θ ) is in fact multilinear.
Revised: . , v L , f ( x ; Θ ) is in fact multilinear.

Original: We also remark that when depth L = 2 , the data tensor is a matrix and the network formulation boils down to f ( x ; Θ ) = v T 1 M ( x ) v 2 .
Revised: We also remark that when depth L = 2 , the data tensor is a matrix and the network formulation boils down to f ( x ; Θ ) = v T 1 M ( x ) v 2 .

Original: Since the data tensor M ( x ) is a linear function of x , the linear tensor network is also a linear function of x .
Revised: Since the data tensor M ( x ) is a linear function of x , the linear tensor network is also a linear function of x .

Original: Thus, the output of the network can also be written as f ( x ; Θ ) =
Revised: Thus, the output of the network can also be written as f ( x ; Θ ) =

Original: x T β ( Θ ) , where β ( Θ ) ∈
Revised: x T β ( Θ ) , where β ( Θ ) ∈

Original: R
Revised: R

Original: d denotes the linear coefﬁcients computed as a function of the network parameters Θ .
Revised: d denotes the linear coefﬁcients computed as a function of the network parameters Θ .

Original: Since the linear tensor network f ( x ; Θ ) is a linear function of x , the expressive power of f is at best a linear model x
Revised: Since the linear tensor network f ( x ; Θ ) is a linear function of x , the expressive power of f is at best a linear model x

Original: (cid:55)→ x T z .
Revised: (cid:55)→ x T z .

Original: However, even though the models have the same expressive power, their architectural differences lead to different implicit biases in training, which we investigate in this paper.
Revised: However, even though the models have the same expressive power, their architectural differences lead to different implicit biases in training, which we investigate in this paper.

Original: Studying separable classiﬁcation and underdetermined regression is useful for highlighting such biases because there are inﬁnitely many coefﬁcients that perfectly classify or ﬁt the dataset.
Revised: Studying separable classiﬁcation and underdetermined regression is useful for highlighting such biases because there are inﬁnitely many coefﬁcients that perfectly classify or ﬁt the dataset.

Original: For our linear tensor network, the evolution of the parameters v l via gradient ﬂow reads
Revised: For our linear tensor network, the evolution of the parameters v l via gradient ﬂow reads

Original: [Equation]
Revised: [Equation]

Original: where we initialize v l (0)
Revised: where we initialize v l (0)

Original: =
Revised: =

Original: α ¯ v l , for l ∈ [ L ] .
Revised: α ¯ v l , for l ∈ [ L ] .

Original: We refer to α and ¯ v l as the initial scale and initial direction , respectively.
Revised: We refer to α and ¯ v l as the initial scale and initial direction , respectively.

Original: The vector r ∈ R n is the residual vector, deﬁned as
Revised: The vector r ∈ R n is the residual vector, deﬁned as

Original: [Equation]
Revised: [Equation]

Original: for classiﬁcation,
Revised: for classiﬁcation,

Original: [Equation]
Revised: [Equation]

Original: for regression.
Revised: for regression.

Original: [Equation]
Revised: [Equation]

Original:  I MPLICIT BIAS OF GRADIENT FLOW IN SEPARABLE CLASSIFICATION
Revised:  I MPLICIT BIAS OF GRADIENT FLOW IN SEPARABLE CLASSIFICATION

Original: In this section, we present our results on the implicit bias of gradient ﬂow in binary classiﬁcation with linearly separable data.
Revised: In this section, we present our results on the implicit bias of gradient ﬂow in binary classiﬁcation with linearly separable data.

Original: Recent papers (Lyu & Li, 2020; Ji & Telgarsky, 2020) on this separable classiﬁcation setup prove that after 100% training accuracy has been achieved by gradient ﬂow (along with other technical conditions), the parameters of L -homogeneous models diverge to inﬁnity, while converging in direction that aligns with the direction of the negative gradient.
Revised: Recent papers (Lyu & Li, 2020; Ji & Telgarsky, 2020) on this separable classiﬁcation setup prove that after 100% training accuracy has been achieved by gradient ﬂow (along with other technical conditions), the parameters of L -homogeneous models diverge to inﬁnity, while converging in direction that aligns with the direction of the negative gradient.

Original: Mathematically,
Revised: Mathematically,

Original: [Equation]
Revised: [Equation]

Original: Since the linear tensor network satisﬁes the technical assumptions in the prior works, we apply these results to our setting and develop a new characterization of the limit directions of the parameters.
Revised: Since the linear tensor network satisﬁes the technical assumptions in the prior works, we apply these results to our setting and develop a new characterization of the limit directions of the parameters.

Original: Here, we present theorems on separable classiﬁcation with general linear tensor networks.
Revised: Here, we present theorems on separable classiﬁcation with general linear tensor networks.

Original: Corollaries for speciﬁc networks are deferred to Appendix A.
Revised: Corollaries for speciﬁc networks are deferred to Appendix A.

Original: 4. L IMIT DIRECTIONS OF PARAMETERS ARE SINGULAR VECTORS
Revised: 4. L IMIT DIRECTIONS OF PARAMETERS ARE SINGULAR VECTORS

Original: Consider the singular value decomposition (SVD) of a matrix
Revised: Consider the singular value decomposition (SVD) of a matrix

Original: A = (cid:80) mj
Revised: A = (cid:80) mj

Original: =

Revised: =

Original: s j ( u j ⊗ v j ) , where m is the rank of A .
Revised: s j ( u j ⊗ v j ) , where m is the rank of A .

Original: Note that the tuples ( u j , v j , s j ) are solutions to the system of equations s u =
Revised: Note that the tuples ( u j , v j , s j ) are solutions to the system of equations s u =

Original: Av and s v =
Revised: Av and s v =

Original: A T u .
Revised: A T u .

Original: Lim (2005) generalizes this deﬁnition of singular vectors and singular values to higher-order tensors: given an order- L tensor A ∈ R k 1 ×···× k L , we deﬁne the singular vectors u 1 , u 2 , . .
Revised: Lim (2005) generalizes this deﬁnition of singular vectors and singular values to higher-order tensors: given an order- L tensor A ∈ R k 1 ×···× k L , we deﬁne the singular vectors u 1 , u 2 , . .

Original: , u L and singular value s to be the solution of the following system of equations:
Revised: , u L and singular value s to be the solution of the following system of equations:

Original: [Equation]
Revised: [Equation]

Original: Using the deﬁnition of the singular vectors of tensors, we can characterize the limit direction of parameters after reaching 100% training accuracy.
Revised: Using the deﬁnition of the singular vectors of tensors, we can characterize the limit direction of parameters after reaching 100% training accuracy.

Original: In Appendix C, we prove the following:
Revised: In Appendix C, we prove the following:

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume that the gradient ﬂow satisﬁes L ( Θ ( t 0 ))
Revised: Assume that the gradient ﬂow satisﬁes L ( Θ ( t 0 ))

Original: <
Revised: <

Original: 1 for some t 0 ≥ 0 and X T r ( t ) converges in direction, say u ∞ :
Revised: 1 for some t 0 ≥ 0 and X T r ( t ) converges in direction, say u ∞ :

Original: =
Revised: =

Original: lim t →∞ X T r ( t ) (cid:107) X T r ( t ) (cid:107) 2 .
Revised: lim t →∞ X T r ( t ) (cid:107) X T r ( t ) (cid:107) 2 .

Original: Then, v 1 , . .
Revised: Then, v 1 , . .

Original: . , v L converge to the singular vectors of M ( − u ∞ ) .
Revised: . , v L converge to the singular vectors of M ( − u ∞ ) .

Original: For this theorem, we make some convergence assumptions, because the network is fully general; this is the only result where we assume convergence.
Revised: For this theorem, we make some convergence assumptions, because the network is fully general; this is the only result where we assume convergence.

Original: It fact, for the special case of linear fullyconnected networks, the directional convergence assumption is not required, and the linear coefﬁcients β fc ( Θ fc ) converge in direction to the (cid:96) 2 max-margin classiﬁer.
Revised: It fact, for the special case of linear fullyconnected networks, the directional convergence assumption is not required, and the linear coefﬁcients β fc ( Θ fc ) converge in direction to the (cid:96) 2 max-margin classiﬁer.

Original: We state this corollary in
Revised: We state this corollary in

Original: Appendix A.1; this result also appears in Ji & Telgarsky (2020), but we provide an alternative proof.
Revised: Appendix A.1; this result also appears in Ji & Telgarsky (2020), but we provide an alternative proof.

Original: 4. L IMIT DIRECTIONS FOR ORTHOGONALLY DECOMPOSABLE NETWORKS
Revised: 4. L IMIT DIRECTIONS FOR ORTHOGONALLY DECOMPOSABLE NETWORKS

Original: Admittedly, Theorem 1 is not a full characterization of the limit directions, because there are usually multiple solutions that satisfy (8).
Revised: Admittedly, Theorem 1 is not a full characterization of the limit directions, because there are usually multiple solutions that satisfy (8).

Original: For example, in case of L = 2 , the data tensor M ( − u ∞ ) is a matrix and the number of possible limit directions (up to scaling) of ( v 1 , v 2 ) is at least the rank of M ( − u ∞ ) .
Revised: For example, in case of L = 2 , the data tensor M ( − u ∞ ) is a matrix and the number of possible limit directions (up to scaling) of ( v 1 , v 2 ) is at least the rank of M ( − u ∞ ) .

Original: Singular vectors of high order tensors are much less understood than the matrix counterparts, and are much harder to deal with.
Revised: Singular vectors of high order tensors are much less understood than the matrix counterparts, and are much harder to deal with.

Original: Although their existence is implied from the variational formulation (Lim, 2005), they are intractable to compute.
Revised: Although their existence is implied from the variational formulation (Lim, 2005), they are intractable to compute.

Original: Testing if a given number is a singular value, approximating the corresponding singular vectors, and computing the best rankapproximation are all NP-hard (Hillar & Lim, 2013); let alone orthogonal decompositions.
Revised: Testing if a given number is a singular value, approximating the corresponding singular vectors, and computing the best rankapproximation are all NP-hard (Hillar & Lim, 2013); let alone orthogonal decompositions.

Original: Given this intractability, it might be reasonable to make some assumptions on the “structure” of the data tensor M ( x ) , so that they are easier to handle.
Revised: Given this intractability, it might be reasonable to make some assumptions on the “structure” of the data tensor M ( x ) , so that they are easier to handle.

Original: The following assumption deﬁnes a class of orthogonally decomposable data tensors, which includes linear diagonal networks and linear full-length convolutional networks as special cases (for the proof, see Appendix D.2 and D.3).
Revised: The following assumption deﬁnes a class of orthogonally decomposable data tensors, which includes linear diagonal networks and linear full-length convolutional networks as special cases (for the proof, see Appendix D.2 and D.3).

Original: Assumption 1.
Revised: Assumption 1.

Original: For the data tensor M ( x ) ∈ R k 1 ×···× k L of a linear tensor network (6) , there exist a full column rank matrix S ∈
Revised: For the data tensor M ( x ) ∈ R k 1 ×···× k L of a linear tensor network (6) , there exist a full column rank matrix S ∈

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: k l ) and matrices
Revised: k l ) and matrices

Original: [Equation]
Revised: [Equation]

Original: such that U Hl
Revised: such that U Hl

Original: U l
Revised: U l

Original: = I m for all l ∈ [ L ] , and the data tensor M ( x ) can be written as
Revised: = I m for all l ∈ [ L ] , and the data tensor M ( x ) can be written as

Original: [Equation]
Revised: [Equation]

Original: In this assumption, we allow U 1 , . .
Revised: In this assumption, we allow U 1 , . .

Original: , U L and S to be complex matrices, although M ( x ) and parameters v l stay real, as deﬁned earlier.
Revised: , U L and S to be complex matrices, although M ( x ) and parameters v l stay real, as deﬁned earlier.

Original: For a complex matrix A , we use A ∗ to denote its entry-wise complex conjugate, A T to denote its transpose (without conjugating), and A H to denote its conjugate transpose.
Revised: For a complex matrix A , we use A ∗ to denote its entry-wise complex conjugate, A T to denote its transpose (without conjugating), and A H to denote its conjugate transpose.

Original: In case of L = 2 , Assumption 1 requires that the data tensor M ( x ) (now a matrix) has singular value decomposition M ( x ) =
Revised: In case of L = 2 , Assumption 1 requires that the data tensor M ( x ) (now a matrix) has singular value decomposition M ( x ) =

Original: U 1 diag( Sx ) U T 2 ; i.e., the left and right singular vectors are independent of x , and the singular values are linear in x .
Revised: U 1 diag( Sx ) U T 2 ; i.e., the left and right singular vectors are independent of x , and the singular values are linear in x .

Original: Using Assumption 1, the following theorem characterizes the limit directions.
Revised: Using Assumption 1, the following theorem characterizes the limit directions.

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose a linear tensor network satisﬁes Assumption 1.
Revised: Suppose a linear tensor network satisﬁes Assumption 1.

Original: If there exists λ > 0 such that the initial directions ¯ v 1 , . .
Revised: If there exists λ > 0 such that the initial directions ¯ v 1 , . .

Original: , ¯ v L of the network parameters satisfy | [ U Tl ¯ v l ] j | 2 −|
Revised: , ¯ v L of the network parameters satisfy | [ U Tl ¯ v l ] j | 2 −|

Original: [ U TL ¯ v L ] j | 2 ≥ λ for all l ∈
Revised: [ U TL ¯ v L ] j | 2 ≥ λ for all l ∈

Original: [ L − 1] and j ∈ [ m ] , then β ( Θ ( t )) converges in a direction that aligns with S T ρ ∞ , where ρ ∞ ∈ C m denotes a stationary point of the following optimization problem
Revised: [ L − 1] and j ∈ [ m ] , then β ( Θ ( t )) converges in a direction that aligns with S T ρ ∞ , where ρ ∞ ∈ C m denotes a stationary point of the following optimization problem

Original: [Equation]
Revised: [Equation]

Original: If S is invertible, then β ( Θ ( t )) converges in a direction that aligns with a stationary point z ∞ of minimize z ∈ R d
Revised: If S is invertible, then β ( Θ ( t )) converges in a direction that aligns with a stationary point z ∞ of minimize z ∈ R d

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 shows that the gradient ﬂow ﬁnds sparse ρ ∞ that minimizes the (cid:96) 2 /L norm in the “singular value space,” where the data points x i are transformed into vectors Sx i consisting of singular values of M ( x i ) .
Revised: Theorem 2 shows that the gradient ﬂow ﬁnds sparse ρ ∞ that minimizes the (cid:96) 2 /L norm in the “singular value space,” where the data points x i are transformed into vectors Sx i consisting of singular values of M ( x i ) .

Original: Also, the proof of Theorem 2 reveals that in case of L = 2 , the parameters v l ( t ) in fact converge to the top singular vectors of the data tensor M ( − X T r ) ; thus, compared to Theorem 1, we have a more complete characterization of “which” singular vectors to converge to.
Revised: Also, the proof of Theorem 2 reveals that in case of L = 2 , the parameters v l ( t ) in fact converge to the top singular vectors of the data tensor M ( − X T r ) ; thus, compared to Theorem 1, we have a more complete characterization of “which” singular vectors to converge to.

Original: The proof of Theorem 2 is in Appendix D. Since the orthogonal decomposition (Assumption 1) of
Revised: The proof of Theorem 2 is in Appendix D. Since the orthogonal decomposition (Assumption 1) of

Original: M ( x ) tells us that the singular vectors M ( x ) in U 1 , . .
Revised: M ( x ) tells us that the singular vectors M ( x ) in U 1 , . .

Original: , U L are independent of x , we can transform the network parameters v l to U Tl v l and show that the network behaves similar to a linear diagonal network.
Revised: , U L are independent of x , we can transform the network parameters v l to U Tl v l and show that the network behaves similar to a linear diagonal network.

Original: This observation comes in handy in the characterization of limit directions.
Revised: This observation comes in handy in the characterization of limit directions.

Original: Remark 1 (Assumptions on initial directions) .
Revised: Remark 1 (Assumptions on initial directions) .

Original: In order to remove the assumption that the loss converges to zero, at least some condition on initialization is necessary , because there are examples showing non-convergence of gradient ﬂow for certain initializations (Bartlett et al., 2018; Arora et al., 2019a).
Revised: In order to remove the assumption that the loss converges to zero, at least some condition on initialization is necessary , because there are examples showing non-convergence of gradient ﬂow for certain initializations (Bartlett et al., 2018; Arora et al., 2019a).

Original: In our theorems, we pose assumptions on initial directions ¯ v l that are sufﬁcient conditions for the loss L ( Θ ( t )) to converge to zero.
Revised: In our theorems, we pose assumptions on initial directions ¯ v l that are sufﬁcient conditions for the loss L ( Θ ( t )) to converge to zero.

Original: Note that λ can be arbitrarily small, so the condition is satisﬁed with probability 1 if we set ¯ v L = 0 and sample ¯ v l ’s from i.i.d.
Revised: Note that λ can be arbitrarily small, so the condition is satisﬁed with probability 1 if we set ¯ v L = 0 and sample ¯ v l ’s from i.i.d.

Original: Gaussian distribution.
Revised: Gaussian distribution.

Original: Setting one layer to zero to prove convergence is also studied in Wu et al.
Revised: Setting one layer to zero to prove convergence is also studied in Wu et al.

Original: We also note that the condition on ¯ v L being “small” can be replaced with any layer; e.g., convergence still holds if | [ U Tl ¯ v l ] j | 2 − | [ U T 1 ¯ v 1 ] j | 2 ≥ λ for all l = 2 , . . .
Revised: We also note that the condition on ¯ v L being “small” can be replaced with any layer; e.g., convergence still holds if | [ U Tl ¯ v l ] j | 2 − | [ U T 1 ¯ v 1 ] j | 2 ≥ λ for all l = 2 , . . .

Original: , L and j ∈ [ m ] .
Revised: , L and j ∈ [ m ] .

Original: Remark 2 (Comparison to existing results) .
Revised: Remark 2 (Comparison to existing results) .

Original: Theorem 2 leads to corollaries (stated in Appendix A.2) on linear diagonal and full-length convolutional networks, showing that diagonal (or convolutional) networks converge to the stationary point of the max-margin problem with respect to the (cid:96) 2 /L norm (or DFT-domain (cid:96) 2 /L norm).
Revised: Theorem 2 leads to corollaries (stated in Appendix A.2) on linear diagonal and full-length convolutional networks, showing that diagonal (or convolutional) networks converge to the stationary point of the max-margin problem with respect to the (cid:96) 2 /L norm (or DFT-domain (cid:96) 2 /L norm).

Original: Although this was already shown in Gunasekar et al. (2018b), note that Theorem 2 does not rely on convergence assumptions, which is different from the prior results.
Revised: Although this was already shown in Gunasekar et al. (2018b), note that Theorem 2 does not rely on convergence assumptions, which is different from the prior results.

Original: Remark 3 (Implications to architecture design) .
Revised: Remark 3 (Implications to architecture design) .

Original: Theorem 2 shows that the gradient ﬂow ﬁnds a solution that is sparse in a “transformed” input space where all data points are transformed with S .
Revised: Theorem 2 shows that the gradient ﬂow ﬁnds a solution that is sparse in a “transformed” input space where all data points are transformed with S .

Original: This implies something interesting about architecture design: if the sparsity of the solution under a certain linear transformation T is needed, one can design a network using Assumption 1 by setting
Revised: This implies something interesting about architecture design: if the sparsity of the solution under a certain linear transformation T is needed, one can design a network using Assumption 1 by setting

Original: S = T .
Revised: S = T .

Original: Training such a network will give us a solution that has the desired sparsity property.
Revised: Training such a network will give us a solution that has the desired sparsity property.

Original: Other than Assumption 1, there is another setting where we can prove a full characterization of limit directions: when there is one data point ( n = 1 ) and the network is 2-layer ( L = 2 ).
Revised: Other than Assumption 1, there is another setting where we can prove a full characterization of limit directions: when there is one data point ( n = 1 ) and the network is 2-layer ( L = 2 ).

Original: This “extremely overparametrized” case is also motivated by an empirical paper (Zhang et al., 2019) which studies generalization performance of different architectures when there is only one training data point.
Revised: This “extremely overparametrized” case is also motivated by an empirical paper (Zhang et al., 2019) which studies generalization performance of different architectures when there is only one training data point.

Original: Theorem 3.
Revised: Theorem 3.

Original: Suppose we have a 2-layer linear tensor network (6) and a single data point ( x , y ) .
Revised: Suppose we have a 2-layer linear tensor network (6) and a single data point ( x , y ) .

Original: Consider the compact SVD M ( x )
Revised: Consider the compact SVD M ( x )

Original: =
Revised: =

Original: U 1 diag( s ) U T 2 , where U 1 ∈ R k 1 × m , U 2 ∈ R k 2 × m , and s ∈ R m for m ≤ min { k 1 , k 2 } .
Revised: U 1 diag( s ) U T 2 , where U 1 ∈ R k 1 × m , U 2 ∈ R k 2 × m , and s ∈ R m for m ≤ min { k 1 , k 2 } .

Original: Let ρ ∞ ∈ R
Revised: Let ρ ∞ ∈ R

Original: m be a solution of the following optimization problem minimize ρ ∈ R m (cid:107) ρ (cid:107)subject to y s T ρ ≥ 1 .
Revised: m be a solution of the following optimization problem minimize ρ ∈ R m (cid:107) ρ (cid:107)subject to y s T ρ ≥ 1 .

Original: Assume that there exists λ > 0 such that the initial directions ¯ v 1 , ¯ v 2 of the network parameters satisfy [ U T 1 ¯ v 1 ] 2 j −
Revised: Assume that there exists λ > 0 such that the initial directions ¯ v 1 , ¯ v 2 of the network parameters satisfy [ U T 1 ¯ v 1 ] 2 j −

Original: [ U T 2 ¯ v 2 ] 2 j ≥ λ for all j ∈ [ m ] .
Revised: [ U T 2 ¯ v 2 ] 2 j ≥ λ for all j ∈ [ m ] .

Original: Then, v 1 and v 2 converge in direction to U 1 η
Revised: Then, v 1 and v 2 converge in direction to U 1 η

Original: ∞and U 2 η ∞ 2 , where | η ∞ 1 | = | η ∞ 2 | =
Revised: ∞and U 2 η ∞ 2 , where | η ∞ 1 | = | η ∞ 2 | =

Original: | ρ ∞ | (cid:12) 1 / 2 , and sign( η ∞ 1 )
Revised: | ρ ∞ | (cid:12) 1 / 2 , and sign( η ∞ 1 )

Original: =
Revised: =

Original: sign( y ) (cid:12) sign( η ∞ 2 ) .
Revised: sign( y ) (cid:12) sign( η ∞ 2 ) .

Original: The proof of Theorem 3 can be found in Appendix E. Since ρ ∞ is the minimum (cid:96) 1 norm solution in the singular value space, the parameters v 1 and v 2 converge in direction to the top singular vectors.
Revised: The proof of Theorem 3 can be found in Appendix E. Since ρ ∞ is the minimum (cid:96) 1 norm solution in the singular value space, the parameters v 1 and v 2 converge in direction to the top singular vectors.

Original: We would like to emphasize that this theorem can be applied to any network architecture that can be represented as a linear tensor network.
Revised: We would like to emphasize that this theorem can be applied to any network architecture that can be represented as a linear tensor network.

Original: Recall that the previous result (Gunasekar et al., 2018b) only considers full-length ﬁlters ( k 1 = d ), hence providing limited insights on networks with small ﬁlters, e.g., k 1 = 2 .
Revised: Recall that the previous result (Gunasekar et al., 2018b) only considers full-length ﬁlters ( k 1 = d ), hence providing limited insights on networks with small ﬁlters, e.g., k 1 = 2 .

Original: In light of this, we present a corollary in Appendix A.3 showing that linear coefﬁcients of convolutional networks converge in direction to a “ﬁltered” version of x .
Revised: In light of this, we present a corollary in Appendix A.3 showing that linear coefﬁcients of convolutional networks converge in direction to a “ﬁltered” version of x .

Original:  I MPLICIT BIAS OF GRADIENT FLOW IN UNDERDETERMINED REGRESSION
Revised:  I MPLICIT BIAS OF GRADIENT FLOW IN UNDERDETERMINED REGRESSION

Original: In Section 4, the limit directions of parameters we characterized do not depend on initialization.
Revised: In Section 4, the limit directions of parameters we characterized do not depend on initialization.

Original: This is due to the fact that the parameters diverge to inﬁnity in separable classiﬁcation problems, so that the initialization becomes unimportant in the limit.
Revised: This is due to the fact that the parameters diverge to inﬁnity in separable classiﬁcation problems, so that the initialization becomes unimportant in the limit.

Original: This is not the case in regression setting, because parameters do not diverge to inﬁnity.
Revised: This is not the case in regression setting, because parameters do not diverge to inﬁnity.

Original: As we show in this section, the limit points are closely tied to initialization, and our analyses characterize the dependency between them.
Revised: As we show in this section, the limit points are closely tied to initialization, and our analyses characterize the dependency between them.

Original: 5. L IMIT POINT CHARACTERIZATION FOR ORTHOGONALLY DECOMPOSABLE NETWORKS
Revised: 5. L IMIT POINT CHARACTERIZATION FOR ORTHOGONALLY DECOMPOSABLE NETWORKS

Original: For the orthogonally decomposable networks satisfying Assumption 1 with real S and U l ’s, we consider how limit points of gradient ﬂow change according to initialization.
Revised: For the orthogonally decomposable networks satisfying Assumption 1 with real S and U l ’s, we consider how limit points of gradient ﬂow change according to initialization.

Original: We consider a speciﬁc initialization scheme that, in the special case of diagonal networks, corresponds to setting w l (0) =
Revised: We consider a speciﬁc initialization scheme that, in the special case of diagonal networks, corresponds to setting w l (0) =

Original: α ¯ w for l ∈
Revised: α ¯ w for l ∈

Original: [ L − 1] and w L (0) = 0 .
Revised: [ L − 1] and w L (0) = 0 .

Original: We use the following lemma on a relevant system of ODEs:
Revised: We use the following lemma on a relevant system of ODEs:

Original: Lemma 4. Consider the system of ODEs, where p, q : R → R :
Revised: Lemma 4. Consider the system of ODEs, where p, q : R → R :

Original: [Equation]
Revised: [Equation]

Original: Then, the solutions p L ( t ) and q L ( t ) are continuous on their maximal interval of existence of the form ( − c, c )
Revised: Then, the solutions p L ( t ) and q L ( t ) are continuous on their maximal interval of existence of the form ( − c, c )

Original: ⊂ R for some c ∈ (0 , ∞ ] .
Revised: ⊂ R for some c ∈ (0 , ∞ ] .

Original: Deﬁne h L ( t ) =
Revised: Deﬁne h L ( t ) =

Original: p L ( t ) L − 1 q L ( t ) ; then, h L ( t ) is odd and strictly increasing, satisfying lim t ↑ c h L ( t )
Revised: p L ( t ) L − 1 q L ( t ) ; then, h L ( t ) is odd and strictly increasing, satisfying lim t ↑ c h L ( t )

Original: = ∞ and lim t ↓− c h L ( t ) =
Revised: = ∞ and lim t ↓− c h L ( t ) =

Original: −∞ .
Revised: −∞ .

Original: Using the function h L ( t ) from Lemma 4, we can obtain the following theorem that characterizes the limit points as the minimizer of a norm-like function Q L,α, ¯ η among the global minima.
Revised: Using the function h L ( t ) from Lemma 4, we can obtain the following theorem that characterizes the limit points as the minimizer of a norm-like function Q L,α, ¯ η among the global minima.

Original: Theorem 5.
Revised: Theorem 5.

Original: Suppose a linear tensor network satisﬁes Assumption 1.
Revised: Suppose a linear tensor network satisﬁes Assumption 1.

Original: Assume further that the matrices U 1 , . .
Revised: Assume further that the matrices U 1 , . .

Original: , U L and S from Assumption 1 are all real matrices.
Revised: , U L and S from Assumption 1 are all real matrices.

Original: For some λ > 0 , choose any vector ¯ η ∈ R m satisfying [¯ η ] 2 j ≥ λ for all j ∈ [ m ] , and choose initial directions ¯ v l
Revised: For some λ > 0 , choose any vector ¯ η ∈ R m satisfying [¯ η ] 2 j ≥ λ for all j ∈ [ m ] , and choose initial directions ¯ v l

Original: = U l ¯ η for l ∈
Revised: = U l ¯ η for l ∈

Original: [ L − 1] and ¯ v L = 0 .
Revised: [ L − 1] and ¯ v L = 0 .

Original: Then, the linear coefﬁcients β ( Θ ( t )) converge to S T ρ ∞ , where ρ ∞ is the solution of minimize ρ ∈ R m
Revised: Then, the linear coefﬁcients β ( Θ ( t )) converge to S T ρ ∞ , where ρ ∞ is the solution of minimize ρ ∈ R m

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: XS
Revised: XS

Original: [Equation]
Revised: [Equation]

Original: where Q L,α, ¯ η : R m → R is a norm-like function deﬁned using H L ( t ) :
Revised: where Q L,α, ¯ η : R m → R is a norm-like function deﬁned using H L ( t ) :

Original: = (cid:82) t 0 h − 1 L ( τ ) dτ .
Revised: = (cid:82) t 0 h − 1 L ( τ ) dτ .

Original: If S is invertible, then β ( Θ ( t )) converges to the solution z ∞ of minimize z ∈ R d
Revised: If S is invertible, then β ( Θ ( t )) converges to the solution z ∞ of minimize z ∈ R d

Original: Q L,α, ¯ η ( S − T z ) subject to
Revised: Q L,α, ¯ η ( S − T z ) subject to

Original: Xz = y .
Revised: Xz = y .

Original: The proofs of Lemma 4 and Theorem 5 are deferred to Appendix F.
Revised: The proofs of Lemma 4 and Theorem 5 are deferred to Appendix F.

Original: Remark 4 (Interpolation between (cid:96) 1 and (cid:96) 2 ) .
Revised: Remark 4 (Interpolation between (cid:96) 1 and (cid:96) 2 ) .

Original: It can be checked that H L ( t ) grows like the absolute value function if t is large, and grows like a quadratic function if t is close to zero.
Revised: It can be checked that H L ( t ) grows like the absolute value function if t is large, and grows like a quadratic function if t is close to zero.

Original: This means that
Revised: This means that

Original: [Equation]
Revised: [Equation]

Original: so Q L,α, ¯ η interpolates between the weighted (cid:96) 1 and weighted (cid:96) 2 norms of ρ .
Revised: so Q L,α, ¯ η interpolates between the weighted (cid:96) 1 and weighted (cid:96) 2 norms of ρ .

Original: Also, the weights in the norm are dependent on the initialization direction ¯ η unless L = 2 and α → 0 .
Revised: Also, the weights in the norm are dependent on the initialization direction ¯ η unless L = 2 and α → 0 .

Original: In general,
Revised: In general,

Original: Q L,α, ¯ η interpolates the standard (cid:96) 1 and (cid:96) 2 norms only if | [¯ η ] j | is the same for all j ∈ [ m ] .
Revised: Q L,α, ¯ η interpolates the standard (cid:96) 1 and (cid:96) 2 norms only if | [¯ η ] j | is the same for all j ∈ [ m ] .

Original: This result is similar to the observations made in Woodworth et al.
Revised: This result is similar to the observations made in Woodworth et al.

Original: (2020) which considers a diagonal network with a “differential” structure f ( x ; w + , w − )
Revised: (2020) which considers a diagonal network with a “differential” structure f ( x ; w + , w − )

Original: = x T ( w (cid:12) L + − w (cid:12) L − ) .
Revised: = x T ( w (cid:12) L + − w (cid:12) L − ) .

Original: In contrast, our results apply to a more general class of networks, without the need to have the differential structure.
Revised: In contrast, our results apply to a more general class of networks, without the need to have the differential structure.

Original: In
Revised: In

Original: Appendix A.4, we state corollaries of Theorem 5 for linear diagonal networks and linear full-length convolutional networks with even data points.
Revised: Appendix A.4, we state corollaries of Theorem 5 for linear diagonal networks and linear full-length convolutional networks with even data points.

Original: There, we also show that deep matrix sensing with commutative sensor matrices (Arora et al., 2019b) is a special case of our setting.
Revised: There, we also show that deep matrix sensing with commutative sensor matrices (Arora et al., 2019b) is a special case of our setting.

Original: Next, we present the regression counterpart of Theorem 3, for 2-layer linear tensor networks with a single data point.
Revised: Next, we present the regression counterpart of Theorem 3, for 2-layer linear tensor networks with a single data point.

Original: For this extremely overparametrized setup, we can fully characterize the limit points as functions of initialization v 1 (0) =
Revised: For this extremely overparametrized setup, we can fully characterize the limit points as functions of initialization v 1 (0) =

Original: α ¯ v 1 and v 2 (0)
Revised: α ¯ v 1 and v 2 (0)

Original: =
Revised: =

Original: α ¯ v 2 , for any linear tensor networks including linear convolutional networks with ﬁlter size smaller than input dimension.
Revised: α ¯ v 2 , for any linear tensor networks including linear convolutional networks with ﬁlter size smaller than input dimension.

Original: Theorem 6.
Revised: Theorem 6.

Original: Suppose we have a 2-layer linear tensor network (6) and a single data point ( x , y ) .
Revised: Suppose we have a 2-layer linear tensor network (6) and a single data point ( x , y ) .

Original: Consider the compact SVD M ( x )
Revised: Consider the compact SVD M ( x )

Original: =
Revised: =

Original: U 1 diag( s ) U T 2 , where U 1 ∈ R k 1 × m , U 2 ∈ R k 2 × m , and s ∈ R m for m ≤ min { k 1 , k 2 } .
Revised: U 1 diag( s ) U T 2 , where U 1 ∈ R k 1 × m , U 2 ∈ R k 2 × m , and s ∈ R m for m ≤ min { k 1 , k 2 } .

Original: Assume that there exists λ > 0 such that the initial directions ¯ v 1 , ¯ vof the network parameters satisfy [ U T 1 ¯ v 1 ] 2 j −
Revised: Assume that there exists λ > 0 such that the initial directions ¯ v 1 , ¯ vof the network parameters satisfy [ U T 1 ¯ v 1 ] 2 j −

Original: [ U T 2 ¯ v 2 ] 2 j ≥ λ for all j ∈ [ m ] .
Revised: [ U T 2 ¯ v 2 ] 2 j ≥ λ for all j ∈ [ m ] .

Original: Then, gradient ﬂow converges to a global minimizer of the loss L , and v 1 ( t ) and v 2 ( t ) converge to the limit points:
Revised: Then, gradient ﬂow converges to a global minimizer of the loss L , and v 1 ( t ) and v 2 ( t ) converge to the limit points:

Original: [Equation]
Revised: [Equation]

Original: where g − 1 is the inverse of the following strictly increasing function
Revised: where g − 1 is the inverse of the following strictly increasing function

Original: [Equation]
Revised: [Equation]

Original: (2[ s ] j ν ) (cid:17) .
Revised: (2[ s ] j ν ) (cid:17) .

Original: The proof can be found in Appendix G. We can observe that as α → 0 , we have g − 1 (cid:0) yα 2 (cid:1) → ∞ , which results in exponentially faster growth of the sinh( · ) and cosh( · ) for the top singular values.
Revised: The proof can be found in Appendix G. We can observe that as α → 0 , we have g − 1 (cid:0) yα 2 (cid:1) → ∞ , which results in exponentially faster growth of the sinh( · ) and cosh( · ) for the top singular values.

Original: As a result, the top singular vectors dominate the limit points v ∞ 1 and v ∞ 2 as α → 0 , and they do not depend on the initial directions ¯ v 1 , ¯ v 2 .
Revised: As a result, the top singular vectors dominate the limit points v ∞ 1 and v ∞ 2 as α → 0 , and they do not depend on the initial directions ¯ v 1 , ¯ v 2 .

Original: Experiment results in Section 6 support this observation.
Revised: Experiment results in Section 6 support this observation.

Original: 5. I MPLICIT BIAS IN FULLY - CONNECTED NETWORKS : THE α → 0 LIMIT
Revised: 5. I MPLICIT BIAS IN FULLY - CONNECTED NETWORKS : THE α → 0 LIMIT

Original: We state our last theoretical element of this paper, which proves that the linear coefﬁcients β fc ( Θ fc ) of deep linear fully-connected networks converge to the minimum (cid:96) 2 norm solution as α → 0 .
Revised: We state our last theoretical element of this paper, which proves that the linear coefﬁcients β fc ( Θ fc ) of deep linear fully-connected networks converge to the minimum (cid:96) 2 norm solution as α → 0 .

Original: We assume for simplicity that d 1 = d 2 = · · · = d L = d in this section, but we can extend it for d l ≥ d without too much difﬁculty.
Revised: We assume for simplicity that d 1 = d 2 = · · · = d L = d in this section, but we can extend it for d l ≥ d without too much difﬁculty.

Original: Recall f fc ( x ; Θ fc ) =
Revised: Recall f fc ( x ; Θ fc ) =

Original: x T W 1 · · · W L − 1 w L .
Revised: x T W 1 · · · W L − 1 w L .

Original: We minimize the training loss L with initialization W l (0) =
Revised: We minimize the training loss L with initialization W l (0) =

Original: α ¯ W l for l ∈
Revised: α ¯ W l for l ∈

Original: [ L − 1] and w L (0) =
Revised: [ L − 1] and w L (0) =

Original: α ¯ w L .
Revised: α ¯ w L .

Original: Theorem 7.
Revised: Theorem 7.

Original: Assume that initial directions ¯ W 1 , . .
Revised: Assume that initial directions ¯ W 1 , . .

Original: , ¯ W L − 1 , ¯ w L satisfy (1) ¯ W Tl ¯ W l (cid:23) ¯ W l
Revised: , ¯ W L − 1 , ¯ w L satisfy (1) ¯ W Tl ¯ W l (cid:23) ¯ W l

Original: +1 ¯ W Tl + for l ∈
Revised: +1 ¯ W Tl + for l ∈

Original: [ L − 2] , and (2) there exists λ > 0 such that ¯ W TL − 1 ¯ W L − 1 − ¯ w L ¯ w TL (cid:23) λ I d .
Revised: [ L − 2] , and (2) there exists λ > 0 such that ¯ W TL − 1 ¯ W L − 1 − ¯ w L ¯ w TL (cid:23) λ I d .

Original: Then, the gradient ﬂow converges to a global minimum, and lim α → 0 lim t →∞ β fc ( Θ fc ( t ))
Revised: Then, the gradient ﬂow converges to a global minimum, and lim α → 0 lim t →∞ β fc ( Θ fc ( t ))

Original: =
Revised: =

Original: X T ( XX T ) − 1 y .
Revised: X T ( XX T ) − 1 y .

Original: The proof is presented in Appendix H. Theorem 7 shows that in the limit α → 0 , linear fullyconnected networks have bias towards the minimum (cid:96) 2 norm solution, regardless of the depth.
Revised: The proof is presented in Appendix H. Theorem 7 shows that in the limit α → 0 , linear fullyconnected networks have bias towards the minimum (cid:96) 2 norm solution, regardless of the depth.

Original: This is consistent with the results shown for classiﬁcation.
Revised: This is consistent with the results shown for classiﬁcation.

Original: We also note that the convergence to a global minimum holds for any α > 0 , and our sufﬁcient conditions ( ¯ W Tl ¯ W l (cid:23) ¯ W l +1 ¯ W Tl +1 and ¯ W TL − 1 ¯ W L − 1 − ¯ w L ¯ w TL (cid:23) λ I d ) for global convergence is a generalization of the zero-asymmetric initialization scheme ( ¯ W 1 = · · · = ¯ W L − 1 = I d and ¯ w L = 0 ) proposed in Wu et al.
Revised: We also note that the convergence to a global minimum holds for any α > 0 , and our sufﬁcient conditions ( ¯ W Tl ¯ W l (cid:23) ¯ W l +1 ¯ W Tl +1 and ¯ W TL − 1 ¯ W L − 1 − ¯ w L ¯ w TL (cid:23) λ I d ) for global convergence is a generalization of the zero-asymmetric initialization scheme ( ¯ W 1 = · · · = ¯ W L − 1 = I d and ¯ w L = 0 ) proposed in Wu et al.

Original:  E XPERIMENTS Regression.
Revised:  E XPERIMENTS Regression.

Original: To fully visualize the trajectory of linear coefﬁcients, we run simple experiments with 2-layer linear fully-connected/diagonal/convolutional networks with a single 2-dimensional data point ( x , y ) =
Revised: To fully visualize the trajectory of linear coefﬁcients, we run simple experiments with 2-layer linear fully-connected/diagonal/convolutional networks with a single 2-dimensional data point ( x , y ) =

Original: For this dataset, the minimum (cid:96) 2 norm solution (corresponding to fully-connected networks) of the regression problem is [0 . 2 0 . 4] , whereas the minimum (cid:96) 1 norm solution (corresponding to diagonal) is [0 0 . 5] and the minimum DFT-domain (cid:96) 1 norm solution (corresponding to convolutional) is [0 .
Revised: For this dataset, the minimum (cid:96) 2 norm solution (corresponding to fully-connected networks) of the regression problem is [0 . 2 0 . 4] , whereas the minimum (cid:96) 1 norm solution (corresponding to diagonal) is [0 0 . 5] and the minimum DFT-domain (cid:96) 1 norm solution (corresponding to convolutional) is [0 .

Original: We randomly pick four directions ¯ z 1 , . . .
Revised: We randomly pick four directions ¯ z 1 , . . .

Original: ¯ z 4 ∈ R 2 , and choose initial directions of the network parameters in a way that their linear coefﬁcients at initialization are exactly β ( Θ (0))
Revised: ¯ z 4 ∈ R 2 , and choose initial directions of the network parameters in a way that their linear coefﬁcients at initialization are exactly β ( Θ (0))

Original: =
Revised: =

Original: α 2 ¯ z j .
Revised: α 2 ¯ z j .

Original: With varying initial scales α ∈ { 0 . 01 , 0 . 5 , 1 } , we run
Revised: With varying initial scales α ∈ { 0 . 01 , 0 . 5 , 1 } , we run

Original: GD with small step size η = 10 − 3 for large enough number of iterations T = 5 × 10 3 .
Revised: GD with small step size η = 10 − 3 for large enough number of iterations T = 5 × 10 3 .

Original: Figures 1 and 2 plot the trajectories of β ( Θ ) (appropriately clipped for visual clarity) as well as the predicted limit points (Theorem 6).
Revised: Figures 1 and 2 plot the trajectories of β ( Θ ) (appropriately clipped for visual clarity) as well as the predicted limit points (Theorem 6).

Original: We observe that even though the networks start at the same linear coefﬁcients α 2 ¯ z j , they evolve differently due to different architectures.
Revised: We observe that even though the networks start at the same linear coefﬁcients α 2 ¯ z j , they evolve differently due to different architectures.

Original: Note that the prediction of limit points is accurate, and the solution found by GD is less dependent on initial directions when α is small.
Revised: Note that the prediction of limit points is accurate, and the solution found by GD is less dependent on initial directions when α is small.

Original: Classiﬁcation.
Revised: Classiﬁcation.

Original: It is shown in the existing works as well as in Section 4 that the limit directions of linear coefﬁcients are independent of the initialization.
Revised: It is shown in the existing works as well as in Section 4 that the limit directions of linear coefﬁcients are independent of the initialization.

Original: Is this also true in practice?
Revised: Is this also true in practice?

Original: To see this, we run a set of toy experiments on classiﬁcation with two data points ( x 1 , y 1 ) =
Revised: To see this, we run a set of toy experiments on classiﬁcation with two data points ( x 1 , y 1 ) =

Original: ([1 2] , +1) and ( x 2 , y 2 ) =
Revised: ([1 2] , +1) and ( x 2 , y 2 ) =

Original: ([0 − 3] , − 1) .
Revised: ([0 − 3] , − 1) .

Original: One can check that the max-margin classiﬁers for this problem are in the same directions to the corresponding min-norm solutions in the regression problem above.
Revised: One can check that the max-margin classiﬁers for this problem are in the same directions to the corresponding min-norm solutions in the regression problem above.

Original: We use the same networks as in regression, and the same set of initial directions satisfying β ( Θ (0))
Revised: We use the same networks as in regression, and the same set of initial directions satisfying β ( Θ (0))

Original: =
Revised: =

Original: α 2 ¯ z j .
Revised: α 2 ¯ z j .

Original: With initial scales α ∈ { 0 . 01 , 0 . 5 , 1 } , we run GD with step size η = 5 × 10 − 4 for T = 

Revised: With initial scales α ∈ { 0 . 01 , 0 . 5 , 1 } , we run GD with step size η = 5 × 10 − 4 for T =

Original: × 10 6 iterations.
Revised: × 10 6 iterations.

Original: All experiments reached L ( Θ ) (cid:46) 10 − 5 at the end.
Revised: All experiments reached L ( Θ ) (cid:46) 10 − 5 at the end.

Original: The trajectories are plotted in Figure 2 in the Appendix.
Revised: The trajectories are plotted in Figure 2 in the Appendix.

Original: We ﬁnd that, in contrast to our theoretical characterization, the actual coefﬁcients are quite dependent on initialization, because we do not train the network all the way to zero loss.
Revised: We ﬁnd that, in contrast to our theoretical characterization, the actual coefﬁcients are quite dependent on initialization, because we do not train the network all the way to zero loss.

Original: This observation is also consistent with a recent analysis (Moroshko et al., 2020) for diagonal networks, and suggests that understanding the behavior of iterates after a ﬁnite number of steps is an important future work.
Revised: This observation is also consistent with a recent analysis (Moroshko et al., 2020) for diagonal networks, and suggests that understanding the behavior of iterates after a ﬁnite number of steps is an important future work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper studies the implicit bias of gradient ﬂow on training linear tensor networks.
Revised: This paper studies the implicit bias of gradient ﬂow on training linear tensor networks.

Original: Under a general tensor formulation of linear networks, we provide theorems characterizing how the network architectures and initializations affect the limit directions/points of gradient ﬂow.
Revised: Under a general tensor formulation of linear networks, we provide theorems characterizing how the network architectures and initializations affect the limit directions/points of gradient ﬂow.

Original: Our work provides a uniﬁed framework that connects multiple existing results on implicit bias of gradient ﬂow as special cases.
Revised: Our work provides a uniﬁed framework that connects multiple existing results on implicit bias of gradient ﬂow as special cases.

Original: A C OROLLARIES ON SPECIFIC NETWORK ARCHITECTURES
Revised: A C OROLLARIES ON SPECIFIC NETWORK ARCHITECTURES

Original: We present corollaries obtained by specializing the theorems in the main text to speciﬁc network architectures.
Revised: We present corollaries obtained by specializing the theorems in the main text to speciﬁc network architectures.

Original: We brieﬂy review the linear neural network architectures studied in this section.
Revised: We brieﬂy review the linear neural network architectures studied in this section.

Original: Linear fully-connected networks.
Revised: Linear fully-connected networks.

Original: An L -layer linear fully-connected network is deﬁned as
Revised: An L -layer linear fully-connected network is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where W l ∈ R d l × d l +1 for l ∈
Revised: where W l ∈ R d l × d l +1 for l ∈

Original: [ L − 1] (we use d 1 = d ) and w L ∈ R d L .
Revised: [ L − 1] (we use d 1 = d ) and w L ∈ R d L .

Original: Linear diagonal networks.
Revised: Linear diagonal networks.

Original: An L -layer linear diagonal network is written as
Revised: An L -layer linear diagonal network is written as

Original: [Equation]
Revised: [Equation]

Original: (11) where w l ∈ R d for l ∈
Revised: (11) where w l ∈ R d for l ∈

Original: [ L ] .
Revised: [ L ] .

Original: Linear (circular) convolutional networks.
Revised: Linear (circular) convolutional networks.

Original: An L -layer linear convolutional network is written as
Revised: An L -layer linear convolutional network is written as

Original: [Equation]
Revised: [Equation]

Original: 12) where w l ∈ R k l with k l ≤ d and k L = d , and (cid:63) deﬁnes the circular convolution: for any a ∈ R d and b ∈ R k ( k ≤ d ), we have a (cid:63) b ∈ R d deﬁned as [ a (cid:63) b ] i = (cid:80) kj =

Revised: 12) where w l ∈ R k l with k l ≤ d and k L = d , and (cid:63) deﬁnes the circular convolution: for any a ∈ R d and b ∈ R k ( k ≤ d ), we have a (cid:63) b ∈ R d deﬁned as [ a (cid:63) b ] i = (cid:80) kj =

Original: [ a ] ( i + j − 1) mod d [ b ] j , for i ∈ [ d ] .
Revised: [ a ] ( i + j − 1) mod d [ b ] j , for i ∈ [ d ] .

Original: In case of k l = d for all l ∈ [ L ] , we refer to this network as full-length convolutional networks.
Revised: In case of k l = d for all l ∈ [ L ] , we refer to this network as full-length convolutional networks.

Original: Deep matrix sensing.
Revised: Deep matrix sensing.

Original: The deep matrix sensing problem considered in Gunasekar et al.
Revised: The deep matrix sensing problem considered in Gunasekar et al.

Original: Arora et al. (2019b) aims to minimize the following problem minimize
Revised: Arora et al. (2019b) aims to minimize the following problem minimize

Original: [Equation]
Revised: [Equation]

Original: where the sensor matrices A 1 , .
Revised: where the sensor matrices A 1 , .

Original: A n ∈ R d × d are symmetric.
Revised: A n ∈ R d × d are symmetric.

Original: Following Gunasekar et al.
Revised: Following Gunasekar et al.

Original: Arora et al. (2019b), we consider sensor matrices A 1 , . . .
Revised: Arora et al. (2019b), we consider sensor matrices A 1 , . . .

Original: , A n ∈ R d × d that commute.
Revised: , A n ∈ R d × d that commute.

Original: To make the problem underdetermined, we assume that n ≤ d , and A i ’s are linearly independent.
Revised: To make the problem underdetermined, we assume that n ≤ d , and A i ’s are linearly independent.

Original: A. C OROLLARY OF T HEOREM
Revised: A. C OROLLARY OF T HEOREM

Original: Corollary 1.
Revised: Corollary 1.

Original: Consider an L -layer linear fully-connected network (10) .
Revised: Consider an L -layer linear fully-connected network (10) .

Original: If the training loss satisﬁes
Revised: If the training loss satisﬁes

Original: L ( Θ fc ( t 0 ))
Revised: L ( Θ fc ( t 0 ))

Original: <
Revised: <

Original: 1 for some t 0 ≥ 0 , then β fc ( Θ fc ( t )) converges in a direction that aligns with the solution of the following optimization problem
Revised: 1 for some t 0 ≥ 0 , then β fc ( Θ fc ( t )) converges in a direction that aligns with the solution of the following optimization problem

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: Corollary 1 shows that whenever the network separates the data correctly, the linear coefﬁcients β fc ( Θ fc ) convergence in direction to the (cid:96) 2 max-margin classiﬁer.
Revised: Corollary 1 shows that whenever the network separates the data correctly, the linear coefﬁcients β fc ( Θ fc ) convergence in direction to the (cid:96) 2 max-margin classiﬁer.

Original: Note that this corollary does not require the directional convergence of X T r , which is different from Theorem 1.
Revised: Note that this corollary does not require the directional convergence of X T r , which is different from Theorem 1.

Original: In fact, this corollary also appears in Ji & Telgarsky (2020), but we provide an alternative proof based on our tensor formulation.
Revised: In fact, this corollary also appears in Ji & Telgarsky (2020), but we provide an alternative proof based on our tensor formulation.

Original: The proof of Corollary 1 can be found in Appendix C.
Revised: The proof of Corollary 1 can be found in Appendix C.

Original: A. C OROLLARIES OF T HEOREM
Revised: A. C OROLLARIES OF T HEOREM

Original: Corollary 2.
Revised: Corollary 2.

Original: Consider an L -layer linear diagonal network (11) .
Revised: Consider an L -layer linear diagonal network (11) .

Original: If there exists λ > 0 such that the initial directions ¯ w 1 , . .
Revised: If there exists λ > 0 such that the initial directions ¯ w 1 , . .

Original: , ¯ w L of the network parameters satisfy [ ¯ w l ] 2 j − [ ¯ w L ] 2 j ≥ λ for all l ∈
Revised: , ¯ w L of the network parameters satisfy [ ¯ w l ] 2 j − [ ¯ w L ] 2 j ≥ λ for all l ∈

Original: [ L − 1] and j ∈ [ d ] , then β diag ( Θ diag ( t )) converges in a direction that aligns with a stationary point z ∞ of minimize z ∈ R d (cid:107) z (cid:107) 2 /L subject to y i x
Revised: [ L − 1] and j ∈ [ d ] , then β diag ( Θ diag ( t )) converges in a direction that aligns with a stationary point z ∞ of minimize z ∈ R d (cid:107) z (cid:107) 2 /L subject to y i x

Original: Ti z ≥ 1 , ∀ i ∈ [ n ] .
Revised: Ti z ≥ 1 , ∀ i ∈ [ n ] .

Original: For full-length convolutional networks, we deﬁne F ∈ C d × d to be the matrix of discrete Fourier transform basis [ F ] j,k = 1 √ d exp( − √− 1 · 2 π ( j − 1)( k − 1) d ) .
Revised: For full-length convolutional networks, we deﬁne F ∈ C d × d to be the matrix of discrete Fourier transform basis [ F ] j,k = 1 √ d exp( − √− 1 · 2 π ( j − 1)( k − 1) d ) .

Original: Note that F ∗ =
Revised: Note that F ∗ =

Original: F − 1 , and both F and F ∗ are symmetric, but not Hermitian.
Revised: F − 1 , and both F and F ∗ are symmetric, but not Hermitian.

Original: Corollary 3.
Revised: Corollary 3.

Original: Consider an L -layer linear full-length convolutional network (12) .
Revised: Consider an L -layer linear full-length convolutional network (12) .

Original: If there exists λ > 0 such that the initial directions ¯ w 1 , . .
Revised: If there exists λ > 0 such that the initial directions ¯ w 1 , . .

Original: , ¯ w L of the network parameters satisfy | [ F ¯ w l ] j | 2 − | [ F ¯ w L ] j | 2 ≥ λ for all l ∈
Revised: , ¯ w L of the network parameters satisfy | [ F ¯ w l ] j | 2 − | [ F ¯ w L ] j | 2 ≥ λ for all l ∈

Original: [ L − 1] and j ∈ [ d ] , then β conv ( Θ conv ( t )) converges in a direction that aligns with a stationary point z ∞ of minimize
Revised: [ L − 1] and j ∈ [ d ] , then β conv ( Θ conv ( t )) converges in a direction that aligns with a stationary point z ∞ of minimize

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: R d
Revised: R d

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: Corollary 2 shows that in the limit, linear diagonal network ﬁnds a sparse solution z that is a stationary point of the (cid:96) 2 /L max-margin classiﬁcation problem.
Revised: Corollary 2 shows that in the limit, linear diagonal network ﬁnds a sparse solution z that is a stationary point of the (cid:96) 2 /L max-margin classiﬁcation problem.

Original: Corollary 3 has a similar conclusion except that the standard (cid:96) 2 /L norm is replaced with DFT-domain (cid:96) 2 /L norm.
Revised: Corollary 3 has a similar conclusion except that the standard (cid:96) 2 /L norm is replaced with DFT-domain (cid:96) 2 /L norm.

Original: By specifying mild conditions on initialization (see Remark 1), these corollaries remove the convergence assumptions required in Gunasekar et al.
Revised: By specifying mild conditions on initialization (see Remark 1), these corollaries remove the convergence assumptions required in Gunasekar et al.

Original: (2018b).
Revised: (2018b).

Original: The proofs of Corollaries 2 and 3 are in Appendix D.
Revised: The proofs of Corollaries 2 and 3 are in Appendix D.

Original: A. C OROLLARY OF T HEOREM
Revised: A. C OROLLARY OF T HEOREM

Original: Recall that Theorem 3 can be applied to any 2-layer networks that can be represented as linear tensor networks.
Revised: Recall that Theorem 3 can be applied to any 2-layer networks that can be represented as linear tensor networks.

Original: Examples include the convolutional networks that are not full-length (i.e., ﬁlter size k 1 < d ), which are not covered by the previous result (Gunasekar et al., 2018b).
Revised: Examples include the convolutional networks that are not full-length (i.e., ﬁlter size k 1 < d ), which are not covered by the previous result (Gunasekar et al., 2018b).

Original: Here, we present the characterization of convergence directions of β conv ( Θ conv ( t )) for small-ﬁlter cases: k 1 = 1 and k 1 = 2 .
Revised: Here, we present the characterization of convergence directions of β conv ( Θ conv ( t )) for small-ﬁlter cases: k 1 = 1 and k 1 = 2 .

Original: Corollary 4.
Revised: Corollary 4.

Original: Consider a 2-layer linear convolutional network (12) with k 1 = 1 and a single data point ( x , y ) .
Revised: Consider a 2-layer linear convolutional network (12) with k 1 = 1 and a single data point ( x , y ) .

Original: If there exists λ > 0 such that the initial directions ¯ w 1 and ¯ w 2 of the network parameters satisfy (cid:107) x (cid:107) 2 ¯ v 21 − ( x T ¯ v 2 ) 2 ≥ (cid:107) x (cid:107) 2 λ , then β conv ( Θ conv ( t )) converges in direction that aligns with y x .
Revised: If there exists λ > 0 such that the initial directions ¯ w 1 and ¯ w 2 of the network parameters satisfy (cid:107) x (cid:107) 2 ¯ v 21 − ( x T ¯ v 2 ) 2 ≥ (cid:107) x (cid:107) 2 λ , then β conv ( Θ conv ( t )) converges in direction that aligns with y x .

Original: Consider a 2-layer linear convolutional network (12) with k 1 = 2 and a single data point ( x , y ) .
Revised: Consider a 2-layer linear convolutional network (12) with k 1 = 2 and a single data point ( x , y ) .

Original: Let ←− x :=
Revised: Let ←− x :=

Original: [[ x ]· · · [ x ] d [ x ] 1 ] , and −→ x :=
Revised: [[ x ]· · · [ x ] d [ x ] 1 ] , and −→ x :=

Original: [[ x ] d [ x ]· · · [ x ] d − 1 ] .
Revised: [[ x ] d [ x ]· · · [ x ] d − 1 ] .

Original: If there exists λ >such that the initial directions ¯ w 1 and ¯ w 2 of the network parameters satisfy
Revised: If there exists λ >such that the initial directions ¯ w 1 and ¯ w 2 of the network parameters satisfy

Original: [Equation]
Revised: [Equation]

Original: then β conv ( Θ conv ( t )) converges in a direction that aligns with a ﬁltered version of x :
Revised: then β conv ( Θ conv ( t )) converges in a direction that aligns with a ﬁltered version of x :

Original: [Equation]
Revised: [Equation]

Original: Corollary 4 shows that if the ﬁlter size is k 1 = 1 , then the limit direction of β conv ( Θ conv ) is always the (cid:96) 2 max-margin classiﬁer.
Revised: Corollary 4 shows that if the ﬁlter size is k 1 = 1 , then the limit direction of β conv ( Θ conv ) is always the (cid:96) 2 max-margin classiﬁer.

Original: Note that this is quite different from the case k 1 = d which converges to the DFT-domain (cid:96) 1 max-margin classiﬁer.
Revised: Note that this is quite different from the case k 1 = d which converges to the DFT-domain (cid:96) 1 max-margin classiﬁer.

Original: However, for 1 < k 1 < d , it is difﬁcult to characterize the limit direction as the max-margin classiﬁer of some common norms.
Revised: However, for 1 < k 1 < d , it is difﬁcult to characterize the limit direction as the max-margin classiﬁer of some common norms.

Original: Rather, the limit directions of β conv ( Θ conv ) correspond to a “ﬁltered” version of the data point, and the weights of the ﬁlter depend on the data point x .
Revised: Rather, the limit directions of β conv ( Θ conv ) correspond to a “ﬁltered” version of the data point, and the weights of the ﬁlter depend on the data point x .

Original: For k 1 = 2 , the ﬁlter is a low-pass ﬁlter if the autocorrelation x T ←− x of x is positive, and high-pass if the autocorrelation is negative.
Revised: For k 1 = 2 , the ﬁlter is a low-pass ﬁlter if the autocorrelation x T ←− x of x is positive, and high-pass if the autocorrelation is negative.

Original: For k 1 > 2 , the ﬁlter weights are more complicated to characterize in terms of x , and the ﬁlter length increases as k 1 increases.
Revised: For k 1 > 2 , the ﬁlter weights are more complicated to characterize in terms of x , and the ﬁlter length increases as k 1 increases.

Original: We prove
Revised: We prove

Original: Corollary 4 in Appendix E.
Revised: Corollary 4 in Appendix E.

Original: A. C OROLLARIES OF T HEOREM
Revised: A. C OROLLARIES OF T HEOREM

Original: In this subsection, we apply Theorem 5 to linear diagonal networks, linear full-length convolutional networks with even data, and deep matrix sensing.
Revised: In this subsection, we apply Theorem 5 to linear diagonal networks, linear full-length convolutional networks with even data, and deep matrix sensing.

Original: The proofs of the corollaries can be found in
Revised: The proofs of the corollaries can be found in

Original: Appendix F.
Revised: Appendix F.

Original: Corollary 5.
Revised: Corollary 5.

Original: Consider an L -layer linear diagonal network (11) .
Revised: Consider an L -layer linear diagonal network (11) .

Original: For some λ > 0 , choose any vector ¯ w ∈ R d satisfying [ ¯ w ] 2 j ≥ λ for all j ∈ [ d ] , and choose initial directions ¯ w l = ¯ w for l ∈
Revised: For some λ > 0 , choose any vector ¯ w ∈ R d satisfying [ ¯ w ] 2 j ≥ λ for all j ∈ [ d ] , and choose initial directions ¯ w l = ¯ w for l ∈

Original: [ L − 1] and ¯ w L = 0 .
Revised: [ L − 1] and ¯ w L = 0 .

Original: Then, the linear coefﬁcients β diag ( Θ diag ( t )) converge to the solution z ∞ of minimize z ∈ R d
Revised: Then, the linear coefﬁcients β diag ( Θ diag ( t )) converge to the solution z ∞ of minimize z ∈ R d

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: Xz = y .
Revised: Xz = y .

Original: Recall that the original statement of Assumption 1 allows the matrices S , U 1 , . .
Revised: Recall that the original statement of Assumption 1 allows the matrices S , U 1 , . .

Original: , U L to be complex, but Theorem 5 poses another assumption that these matrices are real.
Revised: , U L to be complex, but Theorem 5 poses another assumption that these matrices are real.

Original: In applying Theorem 2 to convolutional networks to get Corollary 3, we used the fact that the data tensor M conv ( x ) of a linear full-length convolutional network satisﬁes Assumption 1 with S =
Revised: In applying Theorem 2 to convolutional networks to get Corollary 3, we used the fact that the data tensor M conv ( x ) of a linear full-length convolutional network satisﬁes Assumption 1 with S =

Original: d L − 1 2 F and
Revised: d L − 1 2 F and

Original: U 1 = · · · = U L = F ∗ , where F ∈ C d × d is the matrix of discrete Fourier transform basis [ F ] j,k = 1 √ d exp( − √− 1 · 2 π ( j − 1)( k − 1) d ) and F ∗ is the complex conjugate of F .
Revised: U 1 = · · · = U L = F ∗ , where F ∈ C d × d is the matrix of discrete Fourier transform basis [ F ] j,k = 1 √ d exp( − √− 1 · 2 π ( j − 1)( k − 1) d ) and F ∗ is the complex conjugate of F .

Original: Note that these are complex matrices, so one cannot directly apply Theorem 5 to convolutional networks.
Revised: Note that these are complex matrices, so one cannot directly apply Theorem 5 to convolutional networks.

Original: However, it turns out that if the data and initialization are even, we can derive a corollary for convolutional networks.
Revised: However, it turns out that if the data and initialization are even, we can derive a corollary for convolutional networks.

Original: We say that a vector is even when it satisﬁes the even symmetry, as in even functions.
Revised: We say that a vector is even when it satisﬁes the even symmetry, as in even functions.

Original: More concretely, a vector x ∈ R d is even if [ x ] j +2 =
Revised: More concretely, a vector x ∈ R d is even if [ x ] j +2 =

Original: [ x ] d − j for j = 0 , . . .
Revised: [ x ] d − j for j = 0 , . . .

Original: , (cid:98) d − 32 (cid:99) ; i.e., the vector has the even symmetry around its “origin” [ x ] 1 .
Revised: , (cid:98) d − 32 (cid:99) ; i.e., the vector has the even symmetry around its “origin” [ x ] 1 .

Original: From the deﬁnition of the matrix F ∈ C
Revised: From the deﬁnition of the matrix F ∈ C

Original: d × d , it is straightforward to check that if x is real and even, then its DFT F x is also real and even (see Appendix F. for details).
Revised: d × d , it is straightforward to check that if x is real and even, then its DFT F x is also real and even (see Appendix F. for details).

Original: Corollary 6.
Revised: Corollary 6.

Original: Consider an L -layer linear full-length convolutional network (12) .
Revised: Consider an L -layer linear full-length convolutional network (12) .

Original: Assume that the data points { x i } ni =1 are all even.
Revised: Assume that the data points { x i } ni =1 are all even.

Original: For some λ > 0 , choose any even vector ¯ w satisfying [ F ¯ w ] 2 j ≥ λ for all j ∈ [ d ] , and choose initial directions ¯ w l = ¯ w for l ∈
Revised: For some λ > 0 , choose any even vector ¯ w satisfying [ F ¯ w ] 2 j ≥ λ for all j ∈ [ d ] , and choose initial directions ¯ w l = ¯ w for l ∈

Original: [ L − 1] and ¯ w L = 0 .
Revised: [ L − 1] and ¯ w L = 0 .

Original: Then, the linear coefﬁcients β conv ( Θ conv ( t )) converge to the solution z ∞ of minimize z ∈ R d , even
Revised: Then, the linear coefﬁcients β conv ( Θ conv ( t )) converge to the solution z ∞ of minimize z ∈ R d , even

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: Xz
Revised: Xz

Original: [Equation]
Revised: [Equation]

Original: Corollaries 5 and 6 show that the interpolation between minimum weighted (cid:96) 1 and weighted (cid:96) 2 solutions occurs for diagonal networks, and also for convolutional networks (in DFT domain, with the restriction of even symmetry).
Revised: Corollaries 5 and 6 show that the interpolation between minimum weighted (cid:96) 1 and weighted (cid:96) 2 solutions occurs for diagonal networks, and also for convolutional networks (in DFT domain, with the restriction of even symmetry).

Original: The conclusion of Corollary 5 is similar to the results in Woodworth et al.
Revised: The conclusion of Corollary 5 is similar to the results in Woodworth et al.

Original: (2020), but the network architecture (11) considered in our corollary is a slightly different from the “differential” network f ( x ; w + , w − )
Revised: (2020), but the network architecture (11) considered in our corollary is a slightly different from the “differential” network f ( x ; w + , w − )

Original: = x T ( w (cid:12) L + − w (cid:12) L − ) in Woodworth et al.
Revised: = x T ( w (cid:12) L + − w (cid:12) L − ) in Woodworth et al.

Original: As mentioned in the main text, we can actually show that the matrix sensing result in Arora et al.
Revised: As mentioned in the main text, we can actually show that the matrix sensing result in Arora et al.

Original: (2019b) is a special case of our Theorem 5.
Revised: (2019b) is a special case of our Theorem 5.

Original: Given any symmetric matrix M ∈ R d × d , let eig( M ) ∈ R
Revised: Given any symmetric matrix M ∈ R d × d , let eig( M ) ∈ R

Original: d be the d -dimensional vector of eigenvalues of M .
Revised: d be the d -dimensional vector of eigenvalues of M .

Original: Corollary 7.
Revised: Corollary 7.

Original: Consider the depth-
Revised: Consider the depth-

Original: L deep matrix sensing problem (13) .
Revised: L deep matrix sensing problem (13) .

Original: Let A i ’s be symmetric, and assume A 1 , . . .
Revised: Let A i ’s be symmetric, and assume A 1 , . . .

Original: , A n commute.
Revised: , A n commute.

Original: For α > 0 , choose initialization W l (0)
Revised: For α > 0 , choose initialization W l (0)

Original: =
Revised: =

Original: α
Revised: α

Original: I d for l ∈
Revised: I d for l ∈

Original: [ L − 1] and
Revised: [ L − 1] and

Original: W L (0) = 0 .
Revised: W L (0) = 0 .

Original: Then, the product W L ( t ) · · · W 1 ( t ) converge to the solution M ∞ of minimize M ∈ R d × d , symmetric
Revised: Then, the product W L ( t ) · · · W 1 ( t ) converge to the solution M ∞ of minimize M ∈ R d × d , symmetric

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: ( M )
Revised: ( M )

Original: [Equation]
Revised: [Equation]

Original: Under an additional assumption that A i ’s are positive semideﬁnite, Theorem 2 in Arora et al.
Revised: Under an additional assumption that A i ’s are positive semideﬁnite, Theorem 2 in Arora et al.

Original: (2019b) studies the initialization W l (0)
Revised: (2019b) studies the initialization W l (0)

Original: =
Revised: =

Original: α
Revised: α

Original: I d for all l ∈
Revised: I d for all l ∈

Original: [ L ] , and shows that the limit point of
Revised: [ L ] , and shows that the limit point of

Original: W L . .
Revised: W L . .

Original: W 1 converges to the minimum nuclear norm solution as α → 0 .
Revised: W 1 converges to the minimum nuclear norm solution as α → 0 .

Original: We remove the assumption of positive deﬁniteness of A i ’s and let W L (0) =
Revised: We remove the assumption of positive deﬁniteness of A i ’s and let W L (0) =

Original: 0 , to show a complete characterization of the solution found by gradient ﬂow, which interpolates between the minimum nuclear norm (i.e., Schatten 1-norm) solution (when α → 0 ) and the minimum Frobenius norm (i.e., Schatten 2-norm) solution (when α → ∞ ).
Revised: 0 , to show a complete characterization of the solution found by gradient ﬂow, which interpolates between the minimum nuclear norm (i.e., Schatten 1-norm) solution (when α → 0 ) and the minimum Frobenius norm (i.e., Schatten 2-norm) solution (when α → ∞ ).

Original: B T ENSOR REPRESENTATION OF FULLY - CONNECTED NETWORKS
Revised: B T ENSOR REPRESENTATION OF FULLY - CONNECTED NETWORKS

Original: In Section 3, we only deﬁned the data tensor M fc ( x ) of fully-connected networks for L = 2 .
Revised: In Section 3, we only deﬁned the data tensor M fc ( x ) of fully-connected networks for L = 2 .

Original: Here, we describe an iterative procedure constructing the data tensor for deep fully-connected networks.
Revised: Here, we describe an iterative procedure constructing the data tensor for deep fully-connected networks.

Original: We start with T 1 ( x ) :
Revised: We start with T 1 ( x ) :

Original: = x ∈ R d 1 .
Revised: = x ∈ R d 1 .

Original: Next, deﬁne a block diagonal matrix
Revised: Next, deﬁne a block diagonal matrix

Original: T 2 ( x ) ∈ R d 1 d 2 × d 2 where the “diagonals” [ T 2 ( x )] d 1 ( j − 1)+1:
Revised: T 2 ( x ) ∈ R d 1 d 2 × d 2 where the “diagonals” [ T 2 ( x )] d 1 ( j − 1)+1:

Original: d 1 j,j = T 1 ( x ) for j ∈ [ d 2 ] , while all the other entries are ﬁlled with 0.
Revised: d 1 j,j = T 1 ( x ) for j ∈ [ d 2 ] , while all the other entries are ﬁlled with 0.

Original: We continue this “block diagonal” procedure, as the following.
Revised: We continue this “block diagonal” procedure, as the following.

Original: Having deﬁned T l − 1 ( x ) ∈ R d 1 d 2 ×···× d l − 2 d l − 1 × d l − 1 ,
Revised: Having deﬁned T l − 1 ( x ) ∈ R d 1 d 2 ×···× d l − 2 d l − 1 × d l − 1 ,

Original: Deﬁne T l ( x ) ∈ R d 1 d 2 ×···× d l − 1 d
Revised: Deﬁne T l ( x ) ∈ R d 1 d 2 ×···× d l − 1 d

Original: l
Revised: l

Original: ×
Revised: ×

Original: d l
Revised: d l

Original: Set [ T l ( x )] · ,..., · ,d l − 1 ( j − 1)+1:
Revised: Set [ T l ( x )] · ,..., · ,d l − 1 ( j − 1)+1:

Original: d l − 1 j,j = T l − 1 ( x ) , ∀ j ∈ [ d l ] .
Revised: d l − 1 j,j = T l − 1 ( x ) , ∀ j ∈ [ d l ] .

Original: Set all the remaining entries of T l ( x ) to zero.
Revised: Set all the remaining entries of T l ( x ) to zero.

Original: We repeat this process for l = 2 , . . .
Revised: We repeat this process for l = 2 , . . .

Original: , L , and set M fc ( x ) :
Revised: , L , and set M fc ( x ) :

Original: = T L ( x ) .
Revised: = T L ( x ) .

Original: By deﬁning the parameters of the tensor formulation v l = vec( W l ) for l ∈ [ L − 1] and v L = w L , and using the tensor
Revised: By deﬁning the parameters of the tensor formulation v l = vec( W l ) for l ∈ [ L − 1] and v L = w L , and using the tensor

Original: M ( x ) =
Revised: M ( x ) =

Original: M fc ( x ) , we can check the equivalence of (2) and (5).
Revised: M fc ( x ) , we can check the equivalence of (2) and (5).

Original: C P ROOFS OF T HEOREM 1 AND C OROLLARYC. P ROOF OF T HEOREM
Revised: C P ROOFS OF T HEOREM 1 AND C OROLLARYC. P ROOF OF T HEOREM

Original: The proof of Theorem 1 is outlined as follows.
Revised: The proof of Theorem 1 is outlined as follows.

Original: First, using the directional convergence and alignment results in Ji & Telgarsky (2020), we prove that each of our network parameters v l converges in direction, and it aligns with its corresponding negative gradient −∇ v l L .
Revised: First, using the directional convergence and alignment results in Ji & Telgarsky (2020), we prove that each of our network parameters v l converges in direction, and it aligns with its corresponding negative gradient −∇ v l L .

Original: Then, we prove that the directions of v l ’s are actually singular vectors of M ( − u ∞ ) , where u ∞
Revised: Then, we prove that the directions of v l ’s are actually singular vectors of M ( − u ∞ ) , where u ∞

Original: [Equation]
Revised: [Equation]

Original: Since a linear tensor network is an L -homogeneous polynomial of v 1 , . .
Revised: Since a linear tensor network is an L -homogeneous polynomial of v 1 , . .

Original: . , v L , it satisﬁes the assumptions required for Theorems 3.1 and 4.1 in Ji & Telgarsky (2020).
Revised: . , v L , it satisﬁes the assumptions required for Theorems 3.1 and 4.1 in Ji & Telgarsky (2020).

Original: These theorems imply that if the gradient ﬂow satisﬁes L ( Θ ( t 0 )) < 1 for some t 0 ≥ 0 , then Θ ( t ) converges in direction, and the direction aligns with −∇ Θ L ( Θ ( t )) ; that is,
Revised: These theorems imply that if the gradient ﬂow satisﬁes L ( Θ ( t 0 )) < 1 for some t 0 ≥ 0 , then Θ ( t ) converges in direction, and the direction aligns with −∇ Θ L ( Θ ( t )) ; that is,

Original: [Equation]
Revised: [Equation]

Original: For linear tensor networks (6), the parameter Θ is the concatenation of all parameter vectors v 1 , . .
Revised: For linear tensor networks (6), the parameter Θ is the concatenation of all parameter vectors v 1 , . .

Original: . , v L , so (14) holds for Θ = (cid:2) v T
Revised: . , v L , so (14) holds for Θ = (cid:2) v T

Original: [Equation]
Revised: [Equation]

Original: v TL (cid:3) T .
Revised: v TL (cid:3) T .

Original: Now, recall that by the deﬁnition of the linear tensor network, we have the following gradient ﬂow
Revised: Now, recall that by the deﬁnition of the linear tensor network, we have the following gradient ﬂow

Original: [Equation]
Revised: [Equation]

Original: Note that we can apply this to calculate the rate of growth of (cid:107) v l (cid:107) 22 :
Revised: Note that we can apply this to calculate the rate of growth of (cid:107) v l (cid:107) 22 :

Original: [Equation]
Revised: [Equation]

Original: for any l
Revised: for any l

Original: [Equation]
Revised: [Equation]

Original: so the rate at which (cid:107) v l (cid:107) 22 grows over time is the same for all layers
Revised: so the rate at which (cid:107) v l (cid:107) 22 grows over time is the same for all layers

Original: l ∈
Revised: l ∈

Original: [ L ] .
Revised: [ L ] .

Original: By the deﬁnition of Θ and (14), we have
Revised: By the deﬁnition of Θ and (14), we have

Original: [Equation]
Revised: [Equation]

Original: which then implies
Revised: which then implies

Original: [Equation]
Revised: [Equation]

Original: for all l ∈ [ L ] .
Revised: for all l ∈ [ L ] .

Original: Now, let I l be the set of indices that correspond to the components of v l in Θ .
Revised: Now, let I l be the set of indices that correspond to the components of v l in Θ .

Original: It follows from (14) that
Revised: It follows from (14) that

Original: [Equation]
Revised: [Equation]

Original: thus showing the directional convergence of v l ’s.
Revised: thus showing the directional convergence of v l ’s.

Original: Next, it follows from directional convergence of Θ and its alignment with −∇ Θ L ( Θ ) (14) that ∇ Θ L ( Θ ) also converges in direction, in the opposite direction of Θ .
Revised: Next, it follows from directional convergence of Θ and its alignment with −∇ Θ L ( Θ ) (14) that ∇ Θ L ( Θ ) also converges in direction, in the opposite direction of Θ .

Original: By comparing the components in I l ’s, we get that ∇ v l L ( Θ ) converges in the opposite direction of v l .
Revised: By comparing the components in I l ’s, we get that ∇ v l L ( Θ ) converges in the opposite direction of v l .

Original: For any l ∈ [ L ] , now let v ∞ l :=
Revised: For any l ∈ [ L ] , now let v ∞ l :=

Original: lim t →∞ v l ( t ) (cid:107) v l ( t ) (cid:107) 2 .
Revised: lim t →∞ v l ( t ) (cid:107) v l ( t ) (cid:107) 2 .

Original: Also recall the assumption that X T r ( t ) converges in direction; let the unit vector u ∞ :=
Revised: Also recall the assumption that X T r ( t ) converges in direction; let the unit vector u ∞ :=

Original: lim t →∞ X T r ( t ) (cid:107) X T r ( t ) (cid:107) 2 be the limit direction.
Revised: lim t →∞ X T r ( t ) (cid:107) X T r ( t ) (cid:107) 2 be the limit direction.

Original: By the gradient ﬂow dynamics of v l , we have
Revised: By the gradient ﬂow dynamics of v l , we have

Original: [Equation]
Revised: [Equation]

Original: for all l ∈ [ L ] .
Revised: for all l ∈ [ L ] .

Original: Note that this equation has the same form as (8), the deﬁnition of singular vectors in tensors.
Revised: Note that this equation has the same form as (8), the deﬁnition of singular vectors in tensors.

Original: So this proves that ( v ∞ 1 , . .
Revised: So this proves that ( v ∞ 1 , . .

Original: , v ∞ L ) are singular vectors of M ( − u ∞ ) .
Revised: , v ∞ L ) are singular vectors of M ( − u ∞ ) .

Original: C. P ROOF OF C OROLLARY
Revised: C. P ROOF OF C OROLLARY

Original: The proof proceeds as follows.
Revised: The proof proceeds as follows.

Original: First, we will show using the structure of the data tensor M fc that the limit direction of linear coefﬁcients β fc ( Θ ∞ fc ) is proportional to c u ∞ , where c is a nonzero scalar and u ∞ is the limit direction of X T r .
Revised: First, we will show using the structure of the data tensor M fc that the limit direction of linear coefﬁcients β fc ( Θ ∞ fc ) is proportional to c u ∞ , where c is a nonzero scalar and u ∞ is the limit direction of X T r .

Original: Then, through a closer look at u ∞ and c , we will prove that β fc ( Θ ∞ fc ) is in fact a conic combination of the support vectors (i.e., the data points with the minimum margins).
Revised: Then, through a closer look at u ∞ and c , we will prove that β fc ( Θ ∞ fc ) is in fact a conic combination of the support vectors (i.e., the data points with the minimum margins).

Original: Finally, we will compare β fc ( Θ ∞ fc ) with the KKT conditions of the (cid:96) 2 maxmargin classiﬁcation problem and conclude that β fc ( Θ ∞ fc ) must be in the same direction as the (cid:96)max-margin classiﬁer.
Revised: Finally, we will compare β fc ( Θ ∞ fc ) with the KKT conditions of the (cid:96) 2 maxmargin classiﬁcation problem and conclude that β fc ( Θ ∞ fc ) must be in the same direction as the (cid:96)max-margin classiﬁer.

Original: Due to the way how the data tensor M fc is constructed for fully-connected networks (Appendix B), we always have
Revised: Due to the way how the data tensor M fc is constructed for fully-connected networks (Appendix B), we always have

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: X T r ...
Revised: X T r ...

Original:   , . .
Revised:   , . .

Original: . ,   ...
Revised: . ,   ...

Original: X T r     .
Revised: X T r     .

Original: From Theorem 1, we have directional convergence of v 1 and its alignment with −∇ v 1 L ( Θ fc ) .
Revised: From Theorem 1, we have directional convergence of v 1 and its alignment with −∇ v 1 L ( Θ fc ) .

Original: This means that the limit direction v ∞ 1 , which is a ﬁxed vector, must be also in the span of vectors written above.
Revised: This means that the limit direction v ∞ 1 , which is a ﬁxed vector, must be also in the span of vectors written above.

Original: This implies that X T r must also converge to some direction, say u ∞ :=
Revised: This implies that X T r must also converge to some direction, say u ∞ :=

Original: lim t →∞ X T r ( t ) (cid:107) X T r ( t ) (cid:107) 2 .
Revised: lim t →∞ X T r ( t ) (cid:107) X T r ( t ) (cid:107) 2 .

Original: Now recall the deﬁnition of v 1 in case of the fully-connected network: v 1 = vec( W 1 ) .
Revised: Now recall the deﬁnition of v 1 in case of the fully-connected network: v 1 = vec( W 1 ) .

Original: So, by reshaping v ∞ 1 into its original d 1 × d 2 matrix form W ∞ 1 , we have
Revised: So, by reshaping v ∞ 1 into its original d 1 × d 2 matrix form W ∞ 1 , we have

Original: [Equation]
Revised: [Equation]

Original: for some q ∈ R d 2 .
Revised: for some q ∈ R d 2 .

Original: This implies that the linear coefﬁcients β fc ( Θ fc ) of the network converge in direction to
Revised: This implies that the linear coefﬁcients β fc ( Θ fc ) of the network converge in direction to

Original: [Equation]
Revised: [Equation]

Original: where c is some nonzero real number.
Revised: where c is some nonzero real number.

Original: Let us now take a closer look at the vector u ∞ , the limit direction of X T r .
Revised: Let us now take a closer look at the vector u ∞ , the limit direction of X T r .

Original: Recall from Section 2.
Revised: Recall from Section 2.

Original: that for any i ∈ [ n ] ,
Revised: that for any i ∈ [ n ] ,

Original: [Equation]
Revised: [Equation]

Original: in case of classiﬁcation.
Revised: in case of classiﬁcation.

Original: Recall that (cid:107) β fc ( Θ fc ( t ))
Revised: Recall that (cid:107) β fc ( Θ fc ( t ))

Original: (cid:107) 2 → ∞ while converging to a certain direction β fc ( Θ ∞ fc ) .
Revised: (cid:107) 2 → ∞ while converging to a certain direction β fc ( Θ ∞ fc ) .

Original: This means that if
Revised: This means that if

Original: [Equation]
Revised: [Equation]

Original: for any i, j ∈ [ n ] , then
Revised: for any i, j ∈ [ n ] , then

Original: [Equation]
Revised: [Equation]

Original: Take i to be the index of any support vectors, i.e., any i that attains the minimum y i x Ti β fc ( Θ ∞ fc ) among all data points.
Revised: Take i to be the index of any support vectors, i.e., any i that attains the minimum y i x Ti β fc ( Θ ∞ fc ) among all data points.

Original: Using such an i , the observation (16) implies that lim t →∞ [ r ( t )]
Revised: Using such an i , the observation (16) implies that lim t →∞ [ r ( t )]

Original: j = 0 for any x j that is not a support vector.
Revised: j = 0 for any x j that is not a support vector.

Original: Thus, by the argument above, u ∞ can in fact be written as
Revised: Thus, by the argument above, u ∞ can in fact be written as

Original: [Equation]
Revised: [Equation]

Original: where ν i ≥ 0 for all i ∈ [ n ] , and ν j = 0 for x j ’s that are not support vectors.
Revised: where ν i ≥ 0 for all i ∈ [ n ] , and ν j = 0 for x j ’s that are not support vectors.

Original: Combining (17) and (15),
Revised: Combining (17) and (15),

Original: [Equation]
Revised: [Equation]

Original: Recall that we do not yet know whether c , introduced in (15), is positive or negative; we will now show that c has to be negative.
Revised: Recall that we do not yet know whether c , introduced in (15), is positive or negative; we will now show that c has to be negative.

Original: From Lyu & Li (2020), we know that L ( Θ fc ( t ))
Revised: From Lyu & Li (2020), we know that L ( Θ fc ( t ))

Original: → 0 , which implies that y i x Ti β fc ( Θ ∞ fc ) > 0 for all i ∈ [ n ] .
Revised: → 0 , which implies that y i x Ti β fc ( Θ ∞ fc ) > 0 for all i ∈ [ n ] .

Original: However, if c > 0 , then (18) implies that β fc ( Θ ∞ fc ) is inside a cone K deﬁned as
Revised: However, if c > 0 , then (18) implies that β fc ( Θ ∞ fc ) is inside a cone K deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Note that the polar cone of K , denoted as K ◦ , is
Revised: Note that the polar cone of K , denoted as K ◦ , is

Original: [Equation]
Revised: [Equation]

Original: It is known that K ∩ K ◦ = { 0 } for any convex cone K and its polar cone K ◦ .
Revised: It is known that K ∩ K ◦ = { 0 } for any convex cone K and its polar cone K ◦ .

Original: Therefore, having c > 0 implies that there exists some i ∈ [ n ] such that
Revised: Therefore, having c > 0 implies that there exists some i ∈ [ n ] such that

Original: y i
Revised: y i

Original: x
Revised: x

Original: Ti β fc ( Θ ∞ fc ) < 0 , which contradicts the fact that the loss goes to zero as t → ∞ .
Revised: Ti β fc ( Θ ∞ fc ) < 0 , which contradicts the fact that the loss goes to zero as t → ∞ .

Original: Therefore, c in (15) and (18) must be negative:
Revised: Therefore, c in (15) and (18) must be negative:

Original: [Equation]
Revised: [Equation]

Original: for ν i ≥ 

Revised: for ν i ≥

Original: for all i ∈ [ n ] and ν j = 0 for all x j ’s that are not suport vectors.
Revised: for all i ∈ [ n ] and ν j = 0 for all x j ’s that are not suport vectors.

Original: Finally, compare (19) with the KKT conditions of the following optimization problem: minimize z (cid:107)
Revised: Finally, compare (19) with the KKT conditions of the following optimization problem: minimize z (cid:107)

Original: [Equation]
Revised: [Equation]

Original: subject to y
Revised: subject to y

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: z ≥ 1 , ∀ i ∈ [ n ] .
Revised: z ≥ 1 , ∀ i ∈ [ n ] .

Original: The KKT conditions of this problem are z
Revised: The KKT conditions of this problem are z

Original: [Equation]
Revised: [Equation]

Original: i = µ i y i x i , and µ i ≥ 0 , µ i (1 − y i x Ti z ) = 0 for all i ∈ [ n ] , where µ 1 , . .
Revised: i = µ i y i x i , and µ i ≥ 0 , µ i (1 − y i x Ti z ) = 0 for all i ∈ [ n ] , where µ 1 , . .

Original: . , µ n are the dual variables.
Revised: . , µ n are the dual variables.

Original: Note that this is (up to scaling) satisﬁed by β fc ( Θ ∞ fc ) (19), if we replace µ i ’s with ν i ’s. This ﬁnishes the proof that β fc ( Θ ∞ fc ) is aligned with the (cid:96) 2 max-margin classiﬁer.
Revised: Note that this is (up to scaling) satisﬁed by β fc ( Θ ∞ fc ) (19), if we replace µ i ’s with ν i ’s. This ﬁnishes the proof that β fc ( Θ ∞ fc ) is aligned with the (cid:96) 2 max-margin classiﬁer.

Original: D P ROOFS OF T HEOREM 2 AND C OROLLARIES 2 &D. P ROOF OF T HEOREMD.1. C ONVERGENCE OF LOSS TO ZERO
Revised: D P ROOFS OF T HEOREM 2 AND C OROLLARIES 2 &D. P ROOF OF T HEOREMD.1. C ONVERGENCE OF LOSS TO ZERO

Original: Since Theorem 2 does not assume the existence of t 0 ≥ 0 satisfying L ( Θ ( t 0 )) < 1 , we need to ﬁrst show that given the conditions on initialization, the training loss L ( Θ ( t )) converges to zero.
Revised: Since Theorem 2 does not assume the existence of t 0 ≥ 0 satisfying L ( Θ ( t 0 )) < 1 , we need to ﬁrst show that given the conditions on initialization, the training loss L ( Θ ( t )) converges to zero.

Original: Recall from Section 2.1 that
Revised: Recall from Section 2.1 that

Original: [Equation]
Revised: [Equation]

Original: Applying the structure (9) in Assumption 1, we get
Revised: Applying the structure (9) in Assumption 1, we get

Original: [Equation]
Revised: [Equation]

Original: Left-multiplying U Hl (the conjugate transpose of U l ) to both sides, we get
Revised: Left-multiplying U Hl (the conjugate transpose of U l ) to both sides, we get

Original: [Equation]
Revised: [Equation]

Original: where (cid:81) (cid:12) denotes the product using entry-wise multiplication (cid:12) .
Revised: where (cid:81) (cid:12) denotes the product using entry-wise multiplication (cid:12) .

Original: Now consider the rate of growth for the absolute value squared of the j -th component of U Tl v l : d
Revised: Now consider the rate of growth for the absolute value squared of the j -th component of U Tl v l : d

Original: [Equation]
Revised: [Equation]

Original: any
Revised: any

Original: [Equation]
Revised: [Equation]

Original: [ L ]
Revised: [ L ]

Original: [Equation]
Revised: [Equation]

Original: so for any j ∈ [ m ]
Revised: so for any j ∈ [ m ]

Original: [Equation]
Revised: [Equation]

Original: the squared absolute value of the j -th components in U Tl v l grow at the same rate for each layer l ∈
Revised: the squared absolute value of the j -th components in U Tl v l grow at the same rate for each layer l ∈

Original: [ L ] .
Revised: [ L ] .

Original: This means that the gap between any two different layers stays constant for all t ≥ 0 .
Revised: This means that the gap between any two different layers stays constant for all t ≥ 0 .

Original: Combining this with our conditions on initial directions, we have
Revised: Combining this with our conditions on initial directions, we have

Original: [Equation]
Revised: [Equation]

Original: for any j ∈ [ m ] , l ∈
Revised: for any j ∈ [ m ] , l ∈

Original: [ L − 1] , and t ≥ 0 .
Revised: [ L − 1] , and t ≥ 0 .

Original: This inequality also implies
Revised: This inequality also implies

Original: [Equation]
Revised: [Equation]

Original: Let us now consider the time derivative of L ( Θ ( t )) .
Revised: Let us now consider the time derivative of L ( Θ ( t )) .

Original: We have the following chain of upper bounds on the time derivative:
Revised: We have the following chain of upper bounds on the time derivative:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where (a) used the fact that (cid:107) ˙ v L ( t ) (cid:107) 22 ≥ (cid:107) U L U HL ˙
Revised: where (a) used the fact that (cid:107) ˙ v L ( t ) (cid:107) 22 ≥ (cid:107) U L U HL ˙

Original: v L ( t ) (cid:107) 22 because it is a projection onto a subspace, and (cid:107) U L U HL ˙
Revised: v L ( t ) (cid:107) 22 because it is a projection onto a subspace, and (cid:107) U L U HL ˙

Original: v L ( t ) (cid:107) 

Revised: v L ( t ) (cid:107)

Original: = (cid:107) U HL ˙
Revised: = (cid:107) U HL ˙

Original: v L ( t ) (cid:107) 

Revised: v L ( t ) (cid:107)

Original: because U HL U L
Revised: because U HL U L

Original: = I k L ; (b) is due to (20); (c) is due to (22); and (d) used the fact that S ∈ C m × d is a matrix that has full column rank, so for any z ∈ C d , we can use (cid:107) Sz (cid:107) 2 ≥ s min ( S ) (cid:107) z (cid:107) 2 where s min ( S ) is the minimum singular value of S .
Revised: = I k L ; (b) is due to (20); (c) is due to (22); and (d) used the fact that S ∈ C m × d is a matrix that has full column rank, so for any z ∈ C d , we can use (cid:107) Sz (cid:107) 2 ≥ s min ( S ) (cid:107) z (cid:107) 2 where s min ( S ) is the minimum singular value of S .

Original: We now prove a lower bound on the quantity (cid:107) X T r ( t ) (cid:107) 22 .
Revised: We now prove a lower bound on the quantity (cid:107) X T r ( t ) (cid:107) 22 .

Original: Recall from Section 2.1 the deﬁnition of [ r ( t )]
Revised: Recall from Section 2.1 the deﬁnition of [ r ( t )]

Original: i = − y
Revised: i = − y

Original: i exp( − y i f ( x i ; Θ ( t ))) for classiﬁcation problems.
Revised: i exp( − y i f ( x i ; Θ ( t ))) for classiﬁcation problems.

Original: Also, recall the assumption that the dataset is linearly separable, which means that there exists a unit vector z ∈ R d such that
Revised: Also, recall the assumption that the dataset is linearly separable, which means that there exists a unit vector z ∈ R d such that

Original: [Equation]
Revised: [Equation]

Original: >holds for all i ∈ [ n ] , for some γ > 0 .
Revised: >holds for all i ∈ [ n ] , for some γ > 0 .

Original: Using these,
Revised: Using these,

Original: [Equation]
Revised: [Equation]

Original: Combining this with (23), we get
Revised: Combining this with (23), we get

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Therefore, L ( Θ ( t )) → 0 as t → ∞ .
Revised: Therefore, L ( Θ ( t )) → 0 as t → ∞ .

Original: D.1. C HARACTERIZING THE LIMIT DIRECTION
Revised: D.1. C HARACTERIZING THE LIMIT DIRECTION

Original: Since we proved that L ( Θ ( t )) → 0 , the argument in the proof of Theorem 1 applies to this case, and shows that the parameters v l converge in direction and align with ˙ v l = −∇ v l L ( Θ ) .
Revised: Since we proved that L ( Θ ( t )) → 0 , the argument in the proof of Theorem 1 applies to this case, and shows that the parameters v l converge in direction and align with ˙ v l = −∇ v l L ( Θ ) .

Original: Let v ∞ l :=
Revised: Let v ∞ l :=

Original: lim t →∞ v l ( t ) (cid:107) v l ( t ) (cid:107) 2 be the limit direction of v l .
Revised: lim t →∞ v l ( t ) (cid:107) v l ( t ) (cid:107) 2 be the limit direction of v l .

Original: The remaining steps of the proof are as follows.
Revised: The remaining steps of the proof are as follows.

Original: We ﬁrst prove that SX T r ( t ) converges in direction u ∞ .
Revised: We ﬁrst prove that SX T r ( t ) converges in direction u ∞ .

Original: Using this u ∞ , we derive a number of conditions that has to be satisﬁed by the limit directions of the parameters.
Revised: Using this u ∞ , we derive a number of conditions that has to be satisﬁed by the limit directions of the parameters.

Original: Finally, we compare these conditions with the KKT conditions of the minimization problem, and ﬁnish the proof.
Revised: Finally, we compare these conditions with the KKT conditions of the minimization problem, and ﬁnish the proof.

Original: By Assumption 1, we have
Revised: By Assumption 1, we have

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: Here, we deﬁned ρ := (cid:81) (cid:12) l ∈
Revised: Here, we deﬁned ρ := (cid:81) (cid:12) l ∈

Original: [ L ] U T l v l ∈ C m .
Revised: [ L ] U T l v l ∈ C m .

Original: Since the linear coefﬁcients must be real, we have
Revised: Since the linear coefﬁcients must be real, we have

Original: S T ρ ∈ R d for any real v l ’s. Since v l ’s converge in direction, ρ also converges in direction, to ρ ∞ :
Revised: S T ρ ∈ R d for any real v l ’s. Since v l ’s converge in direction, ρ also converges in direction, to ρ ∞ :

Original: = (cid:81) (cid:12)
Revised: = (cid:81) (cid:12)

Original: l ∈
Revised: l ∈

Original: [ L ] U Tl v ∞ l .
Revised: [ L ] U Tl v ∞ l .

Original: So we can express the limit direction of β ( Θ ) as
Revised: So we can express the limit direction of β ( Θ ) as

Original: [Equation]
Revised: [Equation]

Original: From (20) and alignment of v l and ˙ v l , we have
Revised: From (20) and alignment of v l and ˙ v l , we have

Original: [Equation]
Revised: [Equation]

Original: Since all vectors U Tl v l ( t ) converge in direction, the term SX T r ( t ) should also converge in direction.
Revised: Since all vectors U Tl v l ( t ) converge in direction, the term SX T r ( t ) should also converge in direction.

Original: Let u ∞ :
Revised: Let u ∞ :

Original: =
Revised: =

Original: lim t →∞ SX T r ( t ) (cid:107) SX T r ( t ) (cid:107) 2 .
Revised: lim t →∞ SX T r ( t ) (cid:107) SX T r ( t ) (cid:107) 2 .

Original: One can use the same argument as in Appendix C.2, more speciﬁcally (16) and (17), to show that u ∞ can be written as
Revised: One can use the same argument as in Appendix C.2, more speciﬁcally (16) and (17), to show that u ∞ can be written as

Original: [Equation]
Revised: [Equation]

Original: where ν i ≥ 0 for all i ∈ [ n ] , and ν j = 0 for x j ’s that are not support vectors, i.e., those satisfying y
Revised: where ν i ≥ 0 for all i ∈ [ n ] , and ν j = 0 for x j ’s that are not support vectors, i.e., those satisfying y

Original: [Equation]
Revised: [Equation]

Original: Using u ∞ , we can rewrite (25) as
Revised: Using u ∞ , we can rewrite (25) as

Original: [Equation]
Revised: [Equation]

Original: for all l ∈ [ L ] .
Revised: for all l ∈ [ L ] .

Original: Element-wise multiplying U Tl
Revised: Element-wise multiplying U Tl

Original: v ∞ l to both sides gives
Revised: v ∞ l to both sides gives

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where a (cid:12) b denotes element-wise b -th power of the vector a .
Revised: where a (cid:12) b denotes element-wise b -th power of the vector a .

Original: Since the LHS of (27) is a positive real number, we have
Revised: Since the LHS of (27) is a positive real number, we have

Original: [Equation]
Revised: [Equation]

Original: so using this, (27) becomes
Revised: so using this, (27) becomes

Original: [Equation]
Revised: [Equation]

Original: Now element-wise multiply (29) for all l ∈ [ L ] , then we get
Revised: Now element-wise multiply (29) for all l ∈ [ L ] , then we get

Original: [Equation]
Revised: [Equation]

Original: A close look at (30) reveals that if L ≥ 2 , ρ ∞ and u ∞ must satisfy that
Revised: A close look at (30) reveals that if L ≥ 2 , ρ ∞ and u ∞ must satisfy that

Original: [Equation]
Revised: [Equation]

Original: for all j ∈ [ m ] .
Revised: for all j ∈ [ m ] .

Original: There is another condition that has to be satisﬁed when L = 2 :
Revised: There is another condition that has to be satisﬁed when L = 2 :

Original: [Equation]
Revised: [Equation]

Original: for any j, j (cid:48) ∈ [ m ] ; let us prove why.
Revised: for any j, j (cid:48) ∈ [ m ] ; let us prove why.

Original: First, consider the time derivative of [ ρ ] j =
Revised: First, consider the time derivative of [ ρ ] j =

Original: [ U T 1 v 1 ]
Revised: [ U T 1 v 1 ]

Original: j [ U T 2 v 2 ] j .
Revised: j [ U T 2 v 2 ] j .

Original: [Equation]
Revised: [Equation]

Original: where (a) used (20).
Revised: where (a) used (20).

Original: Now consider
Revised: Now consider

Original: [Equation]
Revised: [Equation]

Original: We want to compare this quantity for different j, j (cid:48) ∈ [ m ] .
Revised: We want to compare this quantity for different j, j (cid:48) ∈ [ m ] .

Original: Before we do that, we take a look at the last term in the RHS of (34).
Revised: Before we do that, we take a look at the last term in the RHS of (34).

Original: Recall from (21)
Revised: Recall from (21)

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: For simplicity
Revised: For simplicity

Original: [Equation]
Revised: [Equation]

Original: let δ j :=
Revised: let δ j :=

Original: | [ U T 1 v 1 (0)]
Revised: | [ U T 1 v 1 (0)]

Original: j | 2 − | [ U T 2 v 2 (0)]
Revised: j | 2 − | [ U T 2 v 2 (0)]

Original: j | 2 .
Revised: j | 2 .

Original: Then, we can use (35) and | [ ρ ( t )]
Revised: Then, we can use (35) and | [ ρ ( t )]

Original: j | = | [ U T 1 v 1 ( t )]
Revised: j | = | [ U T 1 v 1 ( t )]

Original: j ||
Revised: j ||

Original: [ U T 2 v 2 ( t )] j | to show that
Revised: [ U T 2 v 2 ( t )] j | to show that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Suppose that there exists j ∈ [ m ] that satisﬁes | [ ρ ∞ ] j | = 0 but | [ u ∞ ] j | > | [ u ∞ ] j (cid:48) | , for some j (cid:48) ∈
Revised: Suppose that there exists j ∈ [ m ] that satisﬁes | [ ρ ∞ ] j | = 0 but | [ u ∞ ] j | > | [ u ∞ ] j (cid:48) | , for some j (cid:48) ∈

Original: [ m ] satisfying | [ ρ ∞ ] j (cid:48) | (cid:54)
Revised: [ m ] satisfying | [ ρ ∞ ] j (cid:48) | (cid:54)

Original: = 0 .
Revised: = 0 .

Original: Note that having | [ ρ ∞ ] j | = 0 and | [ ρ ∞ ] j
Revised: Note that having | [ ρ ∞ ] j | = 0 and | [ ρ ∞ ] j

Original: [Equation]
Revised: [Equation]

Original: | (cid:54) = 0 implies that | [ ρ ( t )]
Revised: | (cid:54) = 0 implies that | [ ρ ( t )]

Original: j (cid:48) | → ∞ and | [ ρ ( t )]
Revised: j (cid:48) | → ∞ and | [ ρ ( t )]

Original: j | | [ ρ ( t )] j (cid:48) | → 0 .
Revised: j | | [ ρ ( t )] j (cid:48) | → 0 .

Original: We now want to compare the ratio of (34) for j and j (cid:48) .
Revised: We now want to compare the ratio of (34) for j and j (cid:48) .

Original: First, note that
Revised: First, note that

Original: [Equation]
Revised: [Equation]

Original: Next, using | [ ρ ( t )]
Revised: Next, using | [ ρ ( t )]

Original: j | | [ ρ ( t )] j (cid:48) | → 0 and the fact that x (cid:55)→ 2 x 2 + δ x √ x 2 + δ is a decreasing function of x ≥ 0 for any δ > 0 , we have
Revised: j | | [ ρ ( t )] j (cid:48) | → 0 and the fact that x (cid:55)→ 2 x 2 + δ x √ x 2 + δ is a decreasing function of x ≥ 0 for any δ > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: for any t ≥ t 0 , when t 0 is large enough.
Revised: for any t ≥ t 0 , when t 0 is large enough.

Original: Combining (36) and (37) to compare the ratio of (34) for j and j (cid:48) , we get that there exists some t (cid:48) 0 ≥ 0 such that for any t ≥ t (cid:48) 0 , we have
Revised: Combining (36) and (37) to compare the ratio of (34) for j and j (cid:48) , we get that there exists some t (cid:48) 0 ≥ 0 such that for any t ≥ t (cid:48) 0 , we have

Original: [Equation]
Revised: [Equation]

Original: This implies that the ratio of the absolute value of time derivative of [ ρ ( t )]
Revised: This implies that the ratio of the absolute value of time derivative of [ ρ ( t )]

Original: j to the absolute value of current value of [ ρ ( t )]
Revised: j to the absolute value of current value of [ ρ ( t )]

Original: j is strictly bigger than that of [ ρ ( t )] j (cid:48) .
Revised: j is strictly bigger than that of [ ρ ( t )] j (cid:48) .

Original: Moreover, we saw in (33) that the phase of ddt [ ρ ( t )] j converges to that of − [ u ∞ ] ∗ j .
Revised: Moreover, we saw in (33) that the phase of ddt [ ρ ( t )] j converges to that of − [ u ∞ ] ∗ j .

Original: Since this holds for all t ≥ t (cid:48) 0 , (38) results in a growth of | [ ρ ( t )]
Revised: Since this holds for all t ≥ t (cid:48) 0 , (38) results in a growth of | [ ρ ( t )]

Original: j | that is exponentially faster than that of | [ ρ ( t )] j (cid:48) | , so [ ρ ( t )]
Revised: j | that is exponentially faster than that of | [ ρ ( t )] j (cid:48) | , so [ ρ ( t )]

Original: j becomes a dominant component in ρ ( t ) as t → ∞ .
Revised: j becomes a dominant component in ρ ( t ) as t → ∞ .

Original: This contradicts that [ ρ ∞ ] j = 0 , hence the condition (32) has to be satisﬁed.
Revised: This contradicts that [ ρ ∞ ] j = 0 , hence the condition (32) has to be satisﬁed.

Original: So far, we have characterized a number of conditions (26), (28), (31), (32) that have to be satisﬁed by the limit directions u ∞ and ρ ∞ of X T r and ρ .
Revised: So far, we have characterized a number of conditions (26), (28), (31), (32) that have to be satisﬁed by the limit directions u ∞ and ρ ∞ of X T r and ρ .

Original: We now consider the following optimization problem and prove that these conditions are in fact the KKT conditions of the optimization problem.
Revised: We now consider the following optimization problem and prove that these conditions are in fact the KKT conditions of the optimization problem.

Original: Consider minimize
Revised: Consider minimize

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: 1 , ∀ i ∈ [ n ] .
Revised: 1 , ∀ i ∈ [ n ] .

Original: [Equation]
Revised: [Equation]

Original: The KKT conditions of this problem are
Revised: The KKT conditions of this problem are

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: i = µ i y i x i , and µ i ≥ 0 , µ i (1 − y i x Ti S T ρ ) = 0 for all i ∈ [ n ] , where µ 1 , . .
Revised: i = µ i y i x i , and µ i ≥ 0 , µ i (1 − y i x Ti S T ρ ) = 0 for all i ∈ [ n ] , where µ 1 , . .

Original: . , µ n are the dual variables.
Revised: . , µ n are the dual variables.

Original: The symbol ∂ (cid:107)·(cid:107) 2 /L denotes the (local) subdifferential of the (cid:96) 2 /L norm, which can be written as ∂
Revised: The symbol ∂ (cid:107)·(cid:107) 2 /L denotes the (local) subdifferential of the (cid:96) 2 /L norm, which can be written as ∂

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: = { u ∈ C m | | [ u ] j | ≤ 1 for all j ∈ [ m ] , and [ ρ ] j (cid:54) = 

Revised: = { u ∈ C m | | [ u ] j | ≤ 1 for all j ∈ [ m ] , and [ ρ ] j (cid:54) =

Original: = ⇒ [ u ] j = exp( √− 1 arg([ ρ ] j )) } , if L = 2 (in this case ∂ (cid:107) ρ (cid:107) 1 is the global subdifferential), and
Revised: = ⇒ [ u ] j = exp( √− 1 arg([ ρ ] j )) } , if L = 2 (in this case ∂ (cid:107) ρ (cid:107) 1 is the global subdifferential), and

Original: [Equation]
Revised: [Equation]

Original: , if L > 2 .
Revised: , if L > 2 .

Original: By replacing µ i ’s with ν i ’s deﬁned in (26), we can check from (26), (28), (31), (32) that the that ρ ∞ and u ∞ satisfy the KKT conditions up to scaling.
Revised: By replacing µ i ’s with ν i ’s deﬁned in (26), we can check from (26), (28), (31), (32) that the that ρ ∞ and u ∞ satisfy the KKT conditions up to scaling.

Original: Therefore, by (24), β ( Θ ( t )) converges in direction aligned with S T ρ ∞ , where ρ ∞ is again aligned with a stationary point (global minimum in case of L = 2 ) of the optimization problem (39).
Revised: Therefore, by (24), β ( Θ ( t )) converges in direction aligned with S T ρ ∞ , where ρ ∞ is again aligned with a stationary point (global minimum in case of L = 2 ) of the optimization problem (39).

Original: If S is invertible, we can get S − T β ( Θ ∞ )
Revised: If S is invertible, we can get S − T β ( Θ ∞ )

Original: ∝ ρ ∞ .
Revised: ∝ ρ ∞ .

Original: Plugging this into the optimization problem (39) gives the last statement of the theorem.
Revised: Plugging this into the optimization problem (39) gives the last statement of the theorem.

Original: D. P ROOF OF C OROLLARY
Revised: D. P ROOF OF C OROLLARY

Original: It sufﬁces to prove that linear diagonal networks satisfy Assumption 1, with S
Revised: It sufﬁces to prove that linear diagonal networks satisfy Assumption 1, with S

Original: = I d .
Revised: = I d .

Original: The proof is very straightforward, since M diag ( x ) ∈ R
Revised: The proof is very straightforward, since M diag ( x ) ∈ R

Original: d ×···× d has [ M diag ( x )] j,j,...,j
Revised: d ×···× d has [ M diag ( x )] j,j,...,j

Original: =
Revised: =

Original: [ x ] j while all the remaining entries are zero.
Revised: [ x ] j while all the remaining entries are zero.

Original: It is straightforward to verify that M diag ( x ) satisﬁes Assumption 1 with
Revised: It is straightforward to verify that M diag ( x ) satisﬁes Assumption 1 with

Original: S =
Revised: S =

Original: U 1 = · · · = U L = I d .
Revised: U 1 = · · · = U L = I d .

Original: A direct substitution into Theorem 2 gives the corollary.
Revised: A direct substitution into Theorem 2 gives the corollary.

Original: D. P ROOF OF C OROLLARY
Revised: D. P ROOF OF C OROLLARY

Original: For full-length convolutional networks ( k 1 = · · · = k L = d ), we will prove that they satisfy
Revised: For full-length convolutional networks ( k 1 = · · · = k L = d ), we will prove that they satisfy

Original: Assumption 1 with S =
Revised: Assumption 1 with S =

Original: d L − 1 2 F and U 1 = · · · = U L = F ∗ , where F ∈ C d × d is the matrix of discrete Fourier transform basis [ F ] j,k = 1 √ d exp( − √− 1 · 2 π ( j − 1)( k − 1) d ) and F ∗ is the complex conjugate of F .
Revised: d L − 1 2 F and U 1 = · · · = U L = F ∗ , where F ∈ C d × d is the matrix of discrete Fourier transform basis [ F ] j,k = 1 √ d exp( − √− 1 · 2 π ( j − 1)( k − 1) d ) and F ∗ is the complex conjugate of F .

Original: For simplicity of notation, deﬁne ψ = exp( − √− 1 · 2 π d ) .
Revised: For simplicity of notation, deﬁne ψ = exp( − √− 1 · 2 π d ) .

Original: With such matrices S and U 1 , . . .
Revised: With such matrices S and U 1 , . . .

Original: , U L , we can write M ( x ) as
Revised: , U L , we can write M ( x ) as

Original: [Equation]
Revised: [Equation]

Original: where a ⊗ L denotes the L -times tensor product of a .
Revised: where a ⊗ L denotes the L -times tensor product of a .

Original: We will show that M ( x ) =
Revised: We will show that M ( x ) =

Original: M conv ( x ) .
Revised: M conv ( x ) .

Original: For any j 1 , . . .
Revised: For any j 1 , . . .

Original: , j L ∈ [ d ] ,
Revised: , j L ∈ [ d ] ,

Original: [Equation]
Revised: [Equation]

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: Using this, we have
Revised: Using this, we have

Original: [Equation]
Revised: [Equation]

Original: Hence, linear full-length convolutional networks satisfy Assumption 1 with S =
Revised: Hence, linear full-length convolutional networks satisfy Assumption 1 with S =

Original: d L − 1 2 F .
Revised: d L − 1 2 F .

Original: A direct substitution into Theorem 2 and then using the fact that | [ F z ] j | = | [ F ∗ z ] j | for any real vector z ∈ R d gives the corollary.
Revised: A direct substitution into Theorem 2 and then using the fact that | [ F z ] j | = | [ F ∗ z ] j | for any real vector z ∈ R d gives the corollary.

Original: E P ROOFS OF T HEOREM 3 AND C OROLLARYE. P ROOF OF T HEOREME.1. C ONVERGENCE OF LOSS TO ZERO
Revised: E P ROOFS OF T HEOREM 3 AND C OROLLARYE. P ROOF OF T HEOREME.1. C ONVERGENCE OF LOSS TO ZERO

Original: Since Theorem 3 does not assume the existence of t 0 ≥ 0 satisfying L ( Θ ( t 0 )) < 1 , we need to ﬁrst show that given the conditions on initialization, the training loss L ( Θ ( t )) converges to zero.
Revised: Since Theorem 3 does not assume the existence of t 0 ≥ 0 satisfying L ( Θ ( t 0 )) < 1 , we need to ﬁrst show that given the conditions on initialization, the training loss L ( Θ ( t )) converges to zero.

Original: Since
Revised: Since

Original: L = 2 and M ( x ) =
Revised: L = 2 and M ( x ) =

Original: U 1 diag( s ) U T 2 , we can write the gradient ﬂow dynamics from Section 2.1 as
Revised: U 1 diag( s ) U T 2 , we can write the gradient ﬂow dynamics from Section 2.1 as

Original: [Equation]
Revised: [Equation]

Original: where r ( t ) = − y exp( − yf ( x ; Θ ( t ))) is the residual of the data point ( x , y ) .
Revised: where r ( t ) = − y exp( − yf ( x ; Θ ( t ))) is the residual of the data point ( x , y ) .

Original: From (40) we get
Revised: From (40) we get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now consider the rate of growth for the j -th component of U T 1 v 1 squared:
Revised: Now consider the rate of growth for the j -th component of U T 1 v 1 squared:

Original: [Equation]
Revised: [Equation]

Original: So for any j ∈ [ m ]
Revised: So for any j ∈ [ m ]

Original: [Equation]
Revised: [Equation]

Original: [ U T 1 v 1 ] 2 j and [ U T 2 v 2 ] 2 j grow at the same rate.
Revised: [ U T 1 v 1 ] 2 j and [ U T 2 v 2 ] 2 j grow at the same rate.

Original: This means that the gap between the two layers stays constant for all t ≥ 0 .
Revised: This means that the gap between the two layers stays constant for all t ≥ 0 .

Original: Combining this with our conditions on initial directions
Revised: Combining this with our conditions on initial directions

Original: [Equation]
Revised: [Equation]

Original: for any j ∈ [ m ] and t ≥ 0 .
Revised: for any j ∈ [ m ] and t ≥ 0 .

Original: This inequality implies
Revised: This inequality implies

Original: [Equation]
Revised: [Equation]

Original: Let us now consider the time derivative of L ( Θ ( t )) .
Revised: Let us now consider the time derivative of L ( Θ ( t )) .

Original: We have the following chain of upper bounds on the time derivative:
Revised: We have the following chain of upper bounds on the time derivative:

Original: [Equation]
Revised: [Equation]

Original: where (a) used the fact that (cid:107) ˙ v 2 ( t ) (cid:107) 22 ≥ (cid:107)
Revised: where (a) used the fact that (cid:107) ˙ v 2 ( t ) (cid:107) 22 ≥ (cid:107)

Original: U 2 U T 2 ˙ v 2 ( t ) (cid:107) 22 because it is a projection onto a subspace, and (cid:107)
Revised: U 2 U T 2 ˙ v 2 ( t ) (cid:107) 22 because it is a projection onto a subspace, and (cid:107)

Original: U 2 U T 2 ˙ v L ( t ) (cid:107) 

Revised: U 2 U T 2 ˙ v L ( t ) (cid:107)

Original: = (cid:107) U T 2 ˙ v 2 ( t ) (cid:107) 

Revised: = (cid:107) U T 2 ˙ v 2 ( t ) (cid:107)

Original: because U T 2 U 2 = I k 2 ; (b) is due to (41); (c) is due to (44).
Revised: because U T 2 U 2 = I k 2 ; (b) is due to (41); (c) is due to (44).

Original: From this, we get
Revised: From this, we get

Original: [Equation]
Revised: [Equation]

Original: Therefore, L ( Θ ( t )) → 0 as t → ∞ .
Revised: Therefore, L ( Θ ( t )) → 0 as t → ∞ .

Original: E.1. C HARACTERIZING THE LIMIT DIRECTION
Revised: E.1. C HARACTERIZING THE LIMIT DIRECTION

Original: Since we proved that L ( Θ ( t )) → 0 , the argument in the proof of Theorem 1 applies to this case, and shows that the parameters v l converge in direction and align with ˙ v l = −∇ v l L ( Θ ) .
Revised: Since we proved that L ( Θ ( t )) → 0 , the argument in the proof of Theorem 1 applies to this case, and shows that the parameters v l converge in direction and align with ˙ v l = −∇ v l L ( Θ ) .

Original: Let v ∞ l :=
Revised: Let v ∞ l :=

Original: lim t →∞ v l ( t ) (cid:107) v l ( t ) (cid:107) 2 be the limit direction of v l .
Revised: lim t →∞ v l ( t ) (cid:107) v l ( t ) (cid:107) 2 be the limit direction of v l .

Original: As done in the proof of Theorem 2, deﬁne ρ ( t ) =
Revised: As done in the proof of Theorem 2, deﬁne ρ ( t ) =

Original: U T 1 v 1 ( t ) (cid:12) U T 2 v 2 ( t ) and ρ ∞ =
Revised: U T 1 v 1 ( t ) (cid:12) U T 2 v 2 ( t ) and ρ ∞ =

Original: U T 1 v ∞ 1 (cid:12) U T 2 v ∞ 2 .
Revised: U T 1 v ∞ 1 (cid:12) U T 2 v ∞ 2 .

Original: It follows from r ( t ) =
Revised: It follows from r ( t ) =

Original: − y exp( − yf ( x ; Θ ( t ))) that we have sign( r ( t ))
Revised: − y exp( − yf ( x ; Θ ( t ))) that we have sign( r ( t ))

Original: = − sign( y ) .
Revised: = − sign( y ) .

Original: Using this, (41), and alignment of v l and ˙ v l , we have
Revised: Using this, (41), and alignment of v l and ˙ v l , we have

Original: [Equation]
Revised: [Equation]

Original: Element-wise multiplying U T l v ∞ l to both sides gives
Revised: Element-wise multiplying U T l v ∞ l to both sides gives

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since the LHSs are positive and s is positive, the following equations have to be satisﬁed for all j ∈ [ m ] :
Revised: Since the LHSs are positive and s is positive, the following equations have to be satisﬁed for all j ∈ [ m ] :

Original: [Equation]
Revised: [Equation]

Original: Now, multiplying both sides of the two equations (46), we get
Revised: Now, multiplying both sides of the two equations (46), we get

Original: [Equation]
Revised: [Equation]

Original: From (48), ρ ∞ must satisfy that
Revised: From (48), ρ ∞ must satisfy that

Original: [Equation]
Revised: [Equation]

Original: ) for all j, j (cid:48) ∈ [ m ] .
Revised: ) for all j, j (cid:48) ∈ [ m ] .

Original: As in the proof of Theorem 2, there is another condition that has to be satisﬁed:
Revised: As in the proof of Theorem 2, there is another condition that has to be satisﬁed:

Original: [Equation]
Revised: [Equation]

Original: for any j, j (cid:48) ∈ [ m ] ; let us prove why.
Revised: for any j, j (cid:48) ∈ [ m ] ; let us prove why.

Original: First, consider the time derivative of [ ρ ] j =
Revised: First, consider the time derivative of [ ρ ] j =

Original: [ U T 1 v 1 ]
Revised: [ U T 1 v 1 ]

Original: j [ U T 2 v 2 ] j .
Revised: j [ U T 2 v 2 ] j .

Original: [Equation]
Revised: [Equation]

Original: where (a) used (41).
Revised: where (a) used (41).

Original: Now consider
Revised: Now consider

Original: [Equation]
Revised: [Equation]

Original: We want to compare this quantity for different j, j (cid:48) ∈ [ m ] .
Revised: We want to compare this quantity for different j, j (cid:48) ∈ [ m ] .

Original: Before we do that, we take a look at the last term in the RHS of (51).
Revised: Before we do that, we take a look at the last term in the RHS of (51).

Original: Recall from (43) that
Revised: Recall from (43) that

Original: [Equation]
Revised: [Equation]

Original: For simplicity, let δ j :=
Revised: For simplicity, let δ j :=

Original: [ U T 1 v 1 (0)] 2 j − [ U T 2 v 2 (0)] 2 j .
Revised: [ U T 1 v 1 (0)] 2 j − [ U T 2 v 2 (0)] 2 j .

Original: Then, we can use (52) and | [ ρ ( t )]
Revised: Then, we can use (52) and | [ ρ ( t )]

Original: j | = | [ U T 1 v 1 ( t )]
Revised: j | = | [ U T 1 v 1 ( t )]

Original: j ||
Revised: j ||

Original: [ U T 2 v 2 ( t )] j | to show that
Revised: [ U T 2 v 2 ( t )] j | to show that

Original: [Equation]
Revised: [Equation]

Original: Suppose that there exists j ∈ [ m ] that satisﬁes [ ρ ∞ ] j = 0 but | [ s ] j | > | [ s ] j (cid:48) | , for some j (cid:48) ∈
Revised: Suppose that there exists j ∈ [ m ] that satisﬁes [ ρ ∞ ] j = 0 but | [ s ] j | > | [ s ] j (cid:48) | , for some j (cid:48) ∈

Original: [ m ] satisfying [ ρ ∞ ] j (cid:48)
Revised: [ m ] satisfying [ ρ ∞ ] j (cid:48)

Original: (cid:54) = 0 .
Revised: (cid:54) = 0 .

Original: Note that having [ ρ ∞ ] j = 0 and [ ρ ∞ ] j (cid:48) (cid:54) = 

Revised: Note that having [ ρ ∞ ] j = 0 and [ ρ ∞ ] j (cid:48) (cid:54) =

Original: implies that | [ ρ ( t )]
Revised: implies that | [ ρ ( t )]

Original: j (cid:48) | → ∞ and | [ ρ ( t )]
Revised: j (cid:48) | → ∞ and | [ ρ ( t )]

Original: j | | [ ρ ( t )] j (cid:48) | → 0 .
Revised: j | | [ ρ ( t )] j (cid:48) | → 0 .

Original: We now want to compare the ratio of (51) for j and j (cid:48) .
Revised: We now want to compare the ratio of (51) for j and j (cid:48) .

Original: Using | [ ρ ( t )]
Revised: Using | [ ρ ( t )]

Original: j | | [ ρ ( t )] j (cid:48) | → 0 and the fact that x (cid:55)→ 2 x 2 + δ x √ x 2 + δ is a decreasing function of x ≥ 0 for any δ > 0 , we have
Revised: j | | [ ρ ( t )] j (cid:48) | → 0 and the fact that x (cid:55)→ 2 x 2 + δ x √ x 2 + δ is a decreasing function of x ≥ 0 for any δ > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: 53) for any t ≥ t 0 , when t 0 is large enough.
Revised: 53) for any t ≥ t 0 , when t 0 is large enough.

Original: Combining | [ s ] j | | [ s ] j (cid:48) | > 1 and (53) to compare the ratio of (51) for j and j (cid:48) , there exists some t 0 ≥ 

Revised: Combining | [ s ] j | | [ s ] j (cid:48) | > 1 and (53) to compare the ratio of (51) for j and j (cid:48) , there exists some t 0 ≥

Original: such that for any t ≥ t 0 , we have
Revised: such that for any t ≥ t 0 , we have

Original: [Equation]
Revised: [Equation]

Original: This implies that the ratio of the absolute value of time derivative of [ ρ ( t )]
Revised: This implies that the ratio of the absolute value of time derivative of [ ρ ( t )]

Original: j to the absolute value of current value of [ ρ ( t )]
Revised: j to the absolute value of current value of [ ρ ( t )]

Original: j is strictly bigger than that of [ ρ ( t )] j (cid:48) .
Revised: j is strictly bigger than that of [ ρ ( t )] j (cid:48) .

Original: Moreover, by the deﬁnition of r ( t ) , ddt [ ρ ( t )]
Revised: Moreover, by the deﬁnition of r ( t ) , ddt [ ρ ( t )]

Original: j does not change sign over time.
Revised: j does not change sign over time.

Original: Since this holds for all t ≥ t 0 , (54) results in a growth of | [ ρ ( t )]
Revised: Since this holds for all t ≥ t 0 , (54) results in a growth of | [ ρ ( t )]

Original: j | that is exponentially faster than that of | [ ρ ( t )] j (cid:48) | , so [ ρ ( t )]
Revised: j | that is exponentially faster than that of | [ ρ ( t )] j (cid:48) | , so [ ρ ( t )]

Original: j becomes a dominant component in ρ ( t ) as t → ∞ .
Revised: j becomes a dominant component in ρ ( t ) as t → ∞ .

Original: This contradicts that [ ρ ∞ ] j = 0 , hence the condition (50) has to be satisﬁed.
Revised: This contradicts that [ ρ ∞ ] j = 0 , hence the condition (50) has to be satisﬁed.

Original: So far, we have characterized some conditions (47), (49), (50) that have to be satisﬁed by the limit direction ρ ∞ of ρ .
Revised: So far, we have characterized some conditions (47), (49), (50) that have to be satisﬁed by the limit direction ρ ∞ of ρ .

Original: We now consider the following optimization problem and prove that these conditions are in fact the KKT conditions of the optimization problem.
Revised: We now consider the following optimization problem and prove that these conditions are in fact the KKT conditions of the optimization problem.

Original: Consider
Revised: Consider

Original: [Equation]
Revised: [Equation]

Original: The KKT condition of this problem is
Revised: The KKT condition of this problem is

Original: [Equation]
Revised: [Equation]

Original: where the global subdifferential ∂ (cid:107)·(cid:107) 1 is deﬁned as ∂ (cid:107) ρ (cid:107) 

Revised: where the global subdifferential ∂ (cid:107)·(cid:107) 1 is deﬁned as ∂ (cid:107) ρ (cid:107)

Original: = { u ∈ R m | | [ u ] j | ≤ 1 for all j ∈ [ m ] , and [ ρ ] j (cid:54) = 

Revised: = { u ∈ R m | | [ u ] j | ≤ 1 for all j ∈ [ m ] , and [ ρ ] j (cid:54) =

Original: = ⇒ [ u ] j = sign([ ρ ] j ) } .
Revised: = ⇒ [ u ] j = sign([ ρ ] j ) } .

Original: We can check from (47), (49), (50) that the that ρ ∞ satisﬁes the KKT condition up to scaling.
Revised: We can check from (47), (49), (50) that the that ρ ∞ satisﬁes the KKT condition up to scaling.

Original: Now, how do we characterize v ∞ 1 and v ∞ 2 in terms of ρ ∞ ?
Revised: Now, how do we characterize v ∞ 1 and v ∞ 2 in terms of ρ ∞ ?

Original: Let η ∞ 1 :=
Revised: Let η ∞ 1 :=

Original: U T 1 v ∞ 1 and η ∞ 2 :=
Revised: U T 1 v ∞ 1 and η ∞ 2 :=

Original: U T 2 v ∞ 2 .
Revised: U T 2 v ∞ 2 .

Original: Then, v ∞ l = U l η ∞ l = U l U Tl v ∞ l holds because any component orthogonal to the column space of
Revised: Then, v ∞ l = U l η ∞ l = U l U Tl v ∞ l holds because any component orthogonal to the column space of

Original: U l stays unchanged while the component in the column space of U l diverges to inﬁnity.
Revised: U l stays unchanged while the component in the column space of U l diverges to inﬁnity.

Original: By (42), | η ∞ 1 | = | η ∞ 2 | = | ρ ∞ | (cid:12) 1 / 2 .
Revised: By (42), | η ∞ 1 | = | η ∞ 2 | = | ρ ∞ | (cid:12) 1 / 2 .

Original: By (45), we have sign( η ∞ 1 ) =
Revised: By (45), we have sign( η ∞ 1 ) =

Original: sign( y ) (cid:12) sign( η ∞ 2 ) .
Revised: sign( y ) (cid:12) sign( η ∞ 2 ) .

Original: E. P ROOF OF C OROLLARY
Revised: E. P ROOF OF C OROLLARY

Original: The proof of Corollary 4 boils down to characterizing the SVD of M conv ( x ) .
Revised: The proof of Corollary 4 boils down to characterizing the SVD of M conv ( x ) .

Original: E.2. T HE k 1 = 1 CASE
Revised: E.2. T HE k 1 = 1 CASE

Original: First, it is straightforward to check that for L = 2 and k 1 = 1 , we have β conv ( Θ conv )
Revised: First, it is straightforward to check that for L = 2 and k 1 = 1 , we have β conv ( Θ conv )

Original: =
Revised: =

Original: v 1 v 2 .
Revised: v 1 v 2 .

Original: For k 1 = 1 , the data tensor is simply M conv ( x ) = x T .
Revised: For k 1 = 1 , the data tensor is simply M conv ( x ) = x T .

Original: Thus, we have U 1 = 1 , U 2 = x (cid:107) x (cid:107) 2 , and s = (cid:107) x (cid:107) 2 .
Revised: Thus, we have U 1 = 1 , U 2 = x (cid:107) x (cid:107) 2 , and s = (cid:107) x (cid:107) 2 .

Original: Substituting U 

Revised: Substituting U

Original: and U 2 to the theorem gives the condition on initial directions in
Revised: and U 2 to the theorem gives the condition on initial directions in

Original: Corollary 4.
Revised: Corollary 4.

Original: Also, the theorem implies us that the limit direction v ∞ 2 of v 2 satisﬁes v ∞ 2 ∝ y v ∞ 1 x .
Revised: Also, the theorem implies us that the limit direction v ∞ 2 of v 2 satisﬁes v ∞ 2 ∝ y v ∞ 1 x .

Original: Using this, it is easy to check that
Revised: Using this, it is easy to check that

Original: [Equation]
Revised: [Equation]

Original: E.2. T HE k 1 = 2 CASE
Revised: E.2. T HE k 1 = 2 CASE

Original: First, it is straightforward to check that for L = 2 and k 1 = 2 , we have
Revised: First, it is straightforward to check that for L = 2 and k 1 = 2 , we have

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ·   [ v
Revised: ·   [ v

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [ v
Revised: [ v

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: · · · [ v 1 ]    · ·
Revised: · · · [ v 1 ]    · ·

Original: [Equation]
Revised: [Equation]

Original: [ v 1 ][ v 1 ] 
Revised: [ v 1 ][ v 1 ] 

Original: [Equation]
Revised: [Equation]

Original: For k 1 = 2 , by deﬁnition, the data tensor is
Revised: For k 1 = 2 , by deﬁnition, the data tensor is

Original: [Equation]
Revised: [Equation]

Original: , and it is straightforward to check that the SVD of this matrix is
Revised: , and it is straightforward to check that the SVD of this matrix is

Original: M conv
Revised: M conv

Original: [Equation]
Revised: [Equation]

Original: x ) = (cid:20)
Revised: x ) = (cid:20)

Original: [Equation]
Revised: [Equation]

Original: (cid:21) =
Revised: (cid:21) =

Original: (cid:20)
Revised: (cid:20)

Original: [Equation]
Revised: [Equation]

Original: / √
Revised: / √

Original: [Equation]
Revised: [Equation]

Original: / √
Revised: / √

Original: [Equation]
Revised: [Equation]

Original: / √
Revised: / √

Original: [Equation]
Revised: [Equation]

Original: / √(cid:21)   (cid:113) (cid:107) x (cid:107)
Revised: / √(cid:21)   (cid:113) (cid:107) x (cid:107)

Original: 22 + x T ←− x  (cid:113) (cid:107) x (cid:107) 22 − x T ←− x     x T + ←− x T √ 2 √
Revised: 22 + x T ←− x  (cid:113) (cid:107) x (cid:107) 22 − x T ←− x     x T + ←− x T √ 2 √

Original: (cid:107) x (cid:107) 

Revised: (cid:107) x (cid:107)

Original: + x T ←− x x T −←− x T √ 2 √ (cid:107) x (cid:107) 22 − x T ←− x   , so
Revised: + x T ←− x x T −←− x T √ 2 √ (cid:107) x (cid:107) 22 − x T ←− x   , so

Original: U 1 = (cid:20) 1 / √1 / √
Revised: U 1 = (cid:20) 1 / √1 / √

Original: [Equation]
Revised: [Equation]

Original: Substituting U 1 and U 2 to the theorem gives the conditions on initial directions.
Revised: Substituting U 1 and U 2 to the theorem gives the conditions on initial directions.

Original: Also, note that the maximum singular value depends on the sign of x T ←− x .
Revised: Also, note that the maximum singular value depends on the sign of x T ←− x .

Original: Consider the optimization problem in the theorem statement: minimize
Revised: Consider the optimization problem in the theorem statement: minimize

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: If x T ←− x > 0 , then the solution ρ ∞ to this problem is in the direction of [ y 0] .
Revised: If x T ←− x > 0 , then the solution ρ ∞ to this problem is in the direction of [ y 0] .

Original: Therefore, the limit directions v ∞ 1 and v ∞ 2 will be of the form
Revised: Therefore, the limit directions v ∞ 1 and v ∞ 2 will be of the form

Original: [Equation]
Revised: [Equation]

Original: where sign( c 1 ) sign( c 2 ) =
Revised: where sign( c 1 ) sign( c 2 ) =

Original: sign( y ) .
Revised: sign( y ) .

Original: Using (56), it is straightforward to check that
Revised: Using (56), it is straightforward to check that

Original: [Equation]
Revised: [Equation]

Original: Similarly, if x T ←− x < 0 , then the solution ρ ∞ is in the direction of [0 y ] .
Revised: Similarly, if x T ←− x < 0 , then the solution ρ ∞ is in the direction of [0 y ] .

Original: Using (56), we have β conv ( Θ ∞ conv )
Revised: Using (56), we have β conv ( Θ ∞ conv )

Original: ∝ y  
Revised: ∝ y  

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: F P ROOFS OF T HEOREM 5, C OROLLARIES 5, 6 & 7, AND L EMMAF. P ROOF OF L EMMA
Revised: F P ROOFS OF T HEOREM 5, C OROLLARIES 5, 6 & 7, AND L EMMAF. P ROOF OF L EMMA

Original: In this subsection, we restate Lemma 4 and prove it.
Revised: In this subsection, we restate Lemma 4 and prove it.

Original: Lemma 4. Consider the system of ODEs, where p, q : R → R :
Revised: Lemma 4. Consider the system of ODEs, where p, q : R → R :

Original: [Equation]
Revised: [Equation]

Original: Then, the solutions p L ( t ) and q L ( t ) are continuous on their maximal interval of existence of the form ( − c, c )
Revised: Then, the solutions p L ( t ) and q L ( t ) are continuous on their maximal interval of existence of the form ( − c, c )

Original: ⊂ R for some c ∈ (0 , ∞ ] .
Revised: ⊂ R for some c ∈ (0 , ∞ ] .

Original: Deﬁne h L ( t ) =
Revised: Deﬁne h L ( t ) =

Original: p L ( t ) L − 1 q L ( t ) ; then, h L ( t ) is odd and strictly increasing, satisfying lim t ↑ c h L ( t )
Revised: p L ( t ) L − 1 q L ( t ) ; then, h L ( t ) is odd and strictly increasing, satisfying lim t ↑ c h L ( t )

Original: = ∞ and lim t ↓− c h L ( t ) =
Revised: = ∞ and lim t ↓− c h L ( t ) =

Original: −∞ .
Revised: −∞ .

Original: Proof
Revised: Proof

Original: First, continuity of p ( t ) and q ( t ) is straightforward because they are the solution of the
Revised: First, continuity of p ( t ) and q ( t ) is straightforward because they are the solution of the

Original: ODE.
Revised: ODE.

Original: Next, deﬁne ˜ p ( t ) = p ( − t ) and ˜ q ( t ) =
Revised: Next, deﬁne ˜ p ( t ) = p ( − t ) and ˜ q ( t ) =

Original: − q ( − t ) .
Revised: − q ( − t ) .

Original: Then, one can show that ˜ p and ˜ q are also the solution of the ODE because
Revised: Then, one can show that ˜ p and ˜ q are also the solution of the ODE because

Original: [Equation]
Revised: [Equation]

Original: However, by the Picard-Lindel¨of theorem, the solution has to be unique; this means that p ( t ) =
Revised: However, by the Picard-Lindel¨of theorem, the solution has to be unique; this means that p ( t ) =

Original: ˜ p
Revised: ˜ p

Original: ( t ) =
Revised: ( t ) =

Original: p ( − t ) and q ( t ) =
Revised: p ( − t ) and q ( t ) =

Original: ˜ q
Revised: ˜ q

Original: ( t ) =
Revised: ( t ) =

Original: − q ( − t ) , which proves that p is even and q is odd and also implies that the domain of p and q has to be of the form ( − c, c ) (i.e. symmetric around the origin) and
Revised: − q ( − t ) , which proves that p is even and q is odd and also implies that the domain of p and q has to be of the form ( − c, c ) (i.e. symmetric around the origin) and

Original: h = p
Revised: h = p

Original: L − 1 q is odd.
Revised: L − 1 q is odd.

Original: To show that h is strictly increasing, it sufﬁces to show that p and q are both strictly increasing on [0 , c ) .
Revised: To show that h is strictly increasing, it sufﬁces to show that p and q are both strictly increasing on [0 , c ) .

Original: To this end, we show that p ( t ) ≥ 1 for all t ∈ [0 , c ) .
Revised: To this end, we show that p ( t ) ≥ 1 for all t ∈ [0 , c ) .

Original: First, due to the initial condition p (0) =and continuity of p , there exists (cid:15) 1 > 0 such that p ( t ) > 0 for all t ∈ [0 , (cid:15) 1 )
Revised: First, due to the initial condition p (0) =and continuity of p , there exists (cid:15) 1 > 0 such that p ( t ) > 0 for all t ∈ [0 , (cid:15) 1 )

Original: =: I 1 .
Revised: =: I 1 .

Original: This implies that ˙ q ( t ) =
Revised: This implies that ˙ q ( t ) =

Original: p ( t ) L − 1 > 0 for t ∈ I 1 , so q is strictly increasing on I 1 .
Revised: p ( t ) L − 1 > 0 for t ∈ I 1 , so q is strictly increasing on I 1 .

Original: Since q (0) = 0 , we have q ( t ) > 0 for t ∈ I 1 , which then implies that ˙ p ( t ) =
Revised: Since q (0) = 0 , we have q ( t ) > 0 for t ∈ I 1 , which then implies that ˙ p ( t ) =

Original: p ( t ) L − 2 q ( t ) > 0 .
Revised: p ( t ) L − 2 q ( t ) > 0 .

Original: Therefore, p is also strictly increasing on I 1 ; this then means p ( t ) ≥ 1 for t ∈ [0 , (cid:15) 1 ] because p (0) = 1 .
Revised: Therefore, p is also strictly increasing on I 1 ; this then means p ( t ) ≥ 1 for t ∈ [0 , (cid:15) 1 ] because p (0) = 1 .

Original: Now, due to p ( (cid:15) 1 ) ≥and continuity of p , there exists (cid:15) 2 > (cid:15) 1 such that p ( t ) > 0 for all t ∈ [ (cid:15) 1 , (cid:15) 2 )
Revised: Now, due to p ( (cid:15) 1 ) ≥and continuity of p , there exists (cid:15) 2 > (cid:15) 1 such that p ( t ) > 0 for all t ∈ [ (cid:15) 1 , (cid:15) 2 )

Original: =:
Revised: =:

Original: I 2 .
Revised: I 2 .

Original: Using the argument above for I 2 results in p ( t ) ≥ 1 for t ∈ [0 , (cid:15) 2 ] .
Revised: Using the argument above for I 2 results in p ( t ) ≥ 1 for t ∈ [0 , (cid:15) 2 ] .

Original: Repeating this until the end of the domain, we can show that p ( t ) ≥ 1 holds for all t ∈ [0 , c ) .
Revised: Repeating this until the end of the domain, we can show that p ( t ) ≥ 1 holds for all t ∈ [0 , c ) .

Original: By p ≥ 1 , we have ˙ q = p
Revised: By p ≥ 1 , we have ˙ q = p

Original: L − 1 ≥ 1 on [0 , c ) , so q is strictly increasing on [0 , c ) .
Revised: L − 1 ≥ 1 on [0 , c ) , so q is strictly increasing on [0 , c ) .

Original: Also, q ( t ) > 0 on (0 , c ) , so ˙ p = p
Revised: Also, q ( t ) > 0 on (0 , c ) , so ˙ p = p

Original: L − 2 q > 0 on (0 , c ) and p is also strictly increasing on [0 , c ) .
Revised: L − 2 q > 0 on (0 , c ) and p is also strictly increasing on [0 , c ) .

Original: This proves that h is strictly increasing on [0 , c ) , and also on ( − c, c ) by oddity of h .
Revised: This proves that h is strictly increasing on [0 , c ) , and also on ( − c, c ) by oddity of h .

Original: Finally, it is left to show lim t ↑ c h ( t )
Revised: Finally, it is left to show lim t ↑ c h ( t )

Original: = ∞ and lim t ↓− c h ( t ) =
Revised: = ∞ and lim t ↓− c h ( t ) =

Original: −∞ .
Revised: −∞ .

Original: If c < ∞ , then this together with monotonicity implies that the limits hold.
Revised: If c < ∞ , then this together with monotonicity implies that the limits hold.

Original: To see why, suppose c < ∞ and lim t ↑ c h ( t ) < ∞ .
Revised: To see why, suppose c < ∞ and lim t ↑ c h ( t ) < ∞ .

Original: Then, p and q can be extended beyond t ≥ c , which contradicts the fact that ( − c, c ) is the maximal interval of existence of the solution.
Revised: Then, p and q can be extended beyond t ≥ c , which contradicts the fact that ( − c, c ) is the maximal interval of existence of the solution.

Original: Next, consider the case c = ∞ .
Revised: Next, consider the case c = ∞ .

Original: From p ( t ) ≥ 1 , we have ˙ q ( t ) ≥ 1 for t ≥ 0 .
Revised: From p ( t ) ≥ 1 , we have ˙ q ( t ) ≥ 1 for t ≥ 0 .

Original: This implies that q ( t ) ≥ t for t ≥ 0 .
Revised: This implies that q ( t ) ≥ t for t ≥ 0 .

Original: Now, ˙ p ( t ) ≥ p ( t ) L − 2 q ( t ) ≥ t , which gives p ( t ) ≥ t 2 2 + 1 for t ≥ 0 .
Revised: Now, ˙ p ( t ) ≥ p ( t ) L − 2 q ( t ) ≥ t , which gives p ( t ) ≥ t 2 2 + 1 for t ≥ 0 .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: hence ﬁnishing the proof.
Revised: hence ﬁnishing the proof.

Original: F. P ROOF OF T HEOREMF.2. C ONVERGENCE OF LOSS TO ZERO
Revised: F. P ROOF OF T HEOREMF.2. C ONVERGENCE OF LOSS TO ZERO

Original: We ﬁrst show that given the conditions on initialization, the training loss L ( Θ ( t )) converges to zero.
Revised: We ﬁrst show that given the conditions on initialization, the training loss L ( Θ ( t )) converges to zero.

Original: Recall from Section 2.1 that
Revised: Recall from Section 2.1 that

Original: [Equation]
Revised: [Equation]

Original: Applying the structure (9) in Assumption 1, we get ˙
Revised: Applying the structure (9) in Assumption 1, we get ˙

Original: v l = M ( − X T r ) ◦ ( v 1 , . .
Revised: v l = M ( − X T r ) ◦ ( v 1 , . .

Original: . , v l − 1 , I k l , v l +1 , . .
Revised: . , v l − 1 , I k l , v l +1 , . .

Original: . , v L )
Revised: . , v L )

Original: =
Revised: =

Original: − m (cid:88)
Revised: − m (cid:88)

Original: j = [ SX T r ] j ( v T
Revised: j = [ SX T r ] j ( v T

Original: [Equation]
Revised: [Equation]

Original: U
Revised: U

Original: [Equation]
Revised: [Equation]

Original: Left-multiplying U Tl to both sides
Revised: Left-multiplying U Tl to both sides

Original: , we get
Revised: , we get

Original: [Equation]
Revised: [Equation]

Original: where (cid:81) (cid:12) denotes the product using entry-wise multiplication (cid:12) .
Revised: where (cid:81) (cid:12) denotes the product using entry-wise multiplication (cid:12) .

Original: Now consider the rate of growth for the second power of the j -th component of U Tl v l :
Revised: Now consider the rate of growth for the second power of the j -th component of U Tl v l :

Original: [Equation]
Revised: [Equation]

Original: for any l (cid:48) ∈
Revised: for any l (cid:48) ∈

Original: [ L ] .
Revised: [ L ] .

Original: Thus, for any j ∈ [ m ] , the second power of the j -th components in U Tl v l grow at the same rate for each layer l ∈
Revised: Thus, for any j ∈ [ m ] , the second power of the j -th components in U Tl v l grow at the same rate for each layer l ∈

Original: [ L ] .
Revised: [ L ] .

Original: This means that the gap between any two different layers stays constant for all t ≥ 0 .
Revised: This means that the gap between any two different layers stays constant for all t ≥ 0 .

Original: Combining this with our conditions on initial directions, we have
Revised: Combining this with our conditions on initial directions, we have

Original: [Equation]
Revised: [Equation]

Original: for any j ∈ [ m ] , l ∈
Revised: for any j ∈ [ m ] , l ∈

Original: [ L − 1] , and t ≥ 0 .
Revised: [ L − 1] , and t ≥ 0 .

Original: This inequality also implies
Revised: This inequality also implies

Original: [Equation]
Revised: [Equation]

Original: Let us now consider the time derivative of L ( Θ ( t )) .
Revised: Let us now consider the time derivative of L ( Θ ( t )) .

Original: We have the following chain of upper bounds on the time derivative:
Revised: We have the following chain of upper bounds on the time derivative:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (a) used the fact that (cid:107) ˙ v L ( t ) (cid:107) 22 ≥ (cid:107) U L U TL ˙ v L ( t ) (cid:107) 22 because it is a projection onto a subspace, and (cid:107) U L U TL ˙ v L ( t ) (cid:107) 

Revised: where (a) used the fact that (cid:107) ˙ v L ( t ) (cid:107) 22 ≥ (cid:107) U L U TL ˙ v L ( t ) (cid:107) 22 because it is a projection onto a subspace, and (cid:107) U L U TL ˙ v L ( t ) (cid:107)

Original: = (cid:107) U TL ˙ v L ( t ) (cid:107) 

Revised: = (cid:107) U TL ˙ v L ( t ) (cid:107)

Original: because U TL U
Revised: because U TL U

Original: L
Revised: L

Original: = I k L ; (b) is due to (57); (c) is due to (58); and (d) used the fact that S ∈ R m × d and X T ∈ R
Revised: = I k L ; (b) is due to (57); (c) is due to (58); and (d) used the fact that S ∈ R m × d and X T ∈ R

Original: d × n are matrices that have full column rank, so for any z ∈ C n , we can use (cid:107) SX T z (cid:107)
Revised: d × n are matrices that have full column rank, so for any z ∈ C n , we can use (cid:107) SX T z (cid:107)

Original: 2 ≥ s min ( S ) s min ( X ) (cid:107) z (cid:107) 2 where s min ( · ) denotes the minimum singular value of a matrix.
Revised: 2 ≥ s min ( S ) s min ( X ) (cid:107) z (cid:107) 2 where s min ( · ) denotes the minimum singular value of a matrix.

Original: From (59), we get
Revised: From (59), we get

Original: [Equation]
Revised: [Equation]

Original: so that L ( Θ ( t )) → 0 as t → ∞ .
Revised: so that L ( Θ ( t )) → 0 as t → ∞ .

Original: F.2. C HARACTERIZING THE LIMIT POINT
Revised: F.2. C HARACTERIZING THE LIMIT POINT

Original: Now, we move on to characterize the limit points of the gradient ﬂow.
Revised: Now, we move on to characterize the limit points of the gradient ﬂow.

Original: First, by deﬁning a “transformed” version of the parameters η l ( t ) :
Revised: First, by deﬁning a “transformed” version of the parameters η l ( t ) :

Original: = U Tl v l ( t ) and using (57), one can deﬁne an equivalent system of ODEs:
Revised: = U Tl v l ( t ) and using (57), one can deﬁne an equivalent system of ODEs:

Original: [Equation]
Revised: [Equation]

Original: Using Lemma 4, it is straightforward to verify that the solution to (61) has the following form.
Revised: Using Lemma 4, it is straightforward to verify that the solution to (61) has the following form.

Original: For odd L , we have
Revised: For odd L , we have

Original: [Equation]
Revised: [Equation]

Original: Similarly, for even L , the solution for (61) satisﬁes
Revised: Similarly, for even L , the solution for (61) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Now that we know how the solutions η l look like, let us see how these relate to the linear coefﬁcients of the network.
Revised: Now that we know how the solutions η l look like, let us see how these relate to the linear coefﬁcients of the network.

Original: By Assumption 1, we have
Revised: By Assumption 1, we have

Original: [Equation]
Revised: [Equation]

Original: Here, we deﬁned ρ := (cid:81) (cid:12) l ∈
Revised: Here, we deﬁned ρ := (cid:81) (cid:12) l ∈

Original: [ L ] η l ∈ R m .
Revised: [ L ] η l ∈ R m .

Original: Therefore, the linear coefﬁcients of the network can be written as β ( Θ ( t ))
Revised: Therefore, the linear coefﬁcients of the network can be written as β ( Θ ( t ))

Original: =
Revised: =

Original: S T ρ ( t ) .
Revised: S T ρ ( t ) .

Original: From the solutions (62) and (63), we can write
Revised: From the solutions (62) and (63), we can write

Original: [Equation]
Revised: [Equation]

Original: , where h L := p L − 1 L q L , deﬁned in Lemma 4.
Revised: , where h L := p L − 1 L q L , deﬁned in Lemma 4.

Original: By the convergence of the loss to zero (60), we have lim
Revised: By the convergence of the loss to zero (60), we have lim

Original: [Equation]
Revised: [Equation]

Original: ( Θ ( t ))
Revised: ( Θ ( t ))

Original: =
Revised: =

Original: y .
Revised: y .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Next, we will show that ρ ∞ is in fact the solution of the following optimization problem minimize ρ ∈ R m
Revised: Next, we will show that ρ ∞ is in fact the solution of the following optimization problem minimize ρ ∈ R m

Original: Q L,α
Revised: Q L,α

Original: [Equation]
Revised: [Equation]

Original: ¯ η
Revised: ¯ η

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: (65) where Q L,α, ¯ η : R m → R is a norm-like function deﬁned using H L ( t ) :
Revised: (65) where Q L,α, ¯ η : R m → R is a norm-like function deﬁned using H L ( t ) :

Original: = (cid:82) t 0 h − 1 L ( τ ) dτ :
Revised: = (cid:82) t 0 h − 1 L ( τ ) dτ :

Original: Q
Revised: Q

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: Note that the KKT conditions for (65) are
Revised: Note that the KKT conditions for (65) are

Original: [Equation]
Revised: [Equation]

Original: for some ν ∈ R n .
Revised: for some ν ∈ R n .

Original: It is clear from (64) that ρ ∞ satisﬁes the ﬁrst condition (primal feasibility), so let us check the other one.
Revised: It is clear from (64) that ρ ∞ satisﬁes the ﬁrst condition (primal feasibility), so let us check the other one.

Original: Through a straightforward calculation, we get
Revised: Through a straightforward calculation, we get

Original: [Equation]
Revised: [Equation]

Original: Equating this with SX T ν gives
Revised: Equating this with SX T ν gives

Original: [Equation]
Revised: [Equation]

Original: Hence, by setting ν = − (cid:82) ∞ 0 r ( τ ) dτ , ρ ∞ satisﬁes this condition as well.
Revised: Hence, by setting ν = − (cid:82) ∞ 0 r ( τ ) dτ , ρ ∞ satisﬁes this condition as well.

Original: Also, if S is invertible, we can substitute ρ = S − T z to (65) to get the last statement of the theorem.
Revised: Also, if S is invertible, we can substitute ρ = S − T z to (65) to get the last statement of the theorem.

Original: This ﬁnishes the proof.
Revised: This ﬁnishes the proof.

Original: F. P ROOF OF C OROLLARY
Revised: F. P ROOF OF C OROLLARY

Original: The proof is a direct consequence of the fact that Assumption 1 holds with S
Revised: The proof is a direct consequence of the fact that Assumption 1 holds with S

Original: =
Revised: =

Original: U 1 = · · · =
Revised: U 1 = · · · =

Original: U L = I
Revised: U L = I

Original: d for linear diagonal networks.
Revised: d for linear diagonal networks.

Original: Hence, the proof is the same as Corollary 2, proved in
Revised: Hence, the proof is the same as Corollary 2, proved in

Original: Appendix D.2.
Revised: Appendix D.2.

Original: F.
Revised: F.

Original: P
Revised: P

Original: ROOF OF C OROLLARY
Revised: ROOF OF C OROLLARY

Original: We start by showing the DFT of a real and even vector is also real and even.
Revised: We start by showing the DFT of a real and even vector is also real and even.

Original: Suppose that x ∈ R d is real and even.
Revised: Suppose that x ∈ R d is real and even.

Original: First,
Revised: First,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for all j ∈ [ d ] .
Revised: for all j ∈ [ d ] .

Original: To prove that F x is even, for j = 0 , . .
Revised: To prove that F x is even, for j = 0 , . .

Original: , (cid:98) d − 32 (cid:99) , we have
Revised: , (cid:98) d − 32 (cid:99) , we have

Original: [Equation]
Revised: [Equation]

Original: It is proved in Appendix D.3 that linear full-length convolutional networks ( k 1 = · · · = k L = d ) satisfy Assumption 1 with S =
Revised: It is proved in Appendix D.3 that linear full-length convolutional networks ( k 1 = · · · = k L = d ) satisfy Assumption 1 with S =

Original: d L − 1 2 F and U 1 = · · · = U L = F ∗ , where F ∈ C d × d is the matrix of discrete Fourier transform basis [ F ] j,k = 1 √ d exp( − √− 1 · 2 π ( j − 1)( k − 1) d ) and F ∗ is the complex conjugate of F .
Revised: d L − 1 2 F and U 1 = · · · = U L = F ∗ , where F ∈ C d × d is the matrix of discrete Fourier transform basis [ F ] j,k = 1 √ d exp( − √− 1 · 2 π ( j − 1)( k − 1) d ) and F ∗ is the complex conjugate of F .

Original: The proof of convergence of loss to zero in Appendix F.2.1 is written for real matrices
Revised: The proof of convergence of loss to zero in Appendix F.2.1 is written for real matrices

Original: S , U 1 , . . .
Revised: S , U 1 , . . .

Original: , U L , but we can actually apply the same argument as in Appendix D.1.1 and prove that the loss converges to zero, even in the case where S , U 1 , . . .
Revised: , U L , but we can actually apply the same argument as in Appendix D.1.1 and prove that the loss converges to zero, even in the case where S , U 1 , . . .

Original: , U L are complex.
Revised: , U L are complex.

Original: Next, since U l ’s are complex, we can write the system of ODE as (see (20) for its derivation)
Revised: Next, since U l ’s are complex, we can write the system of ODE as (see (20) for its derivation)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since all data points x i and initialization w l (0) are real and even, we have that F X T r is real and even, and F ∗ w l (0)
Revised: Since all data points x i and initialization w l (0) are real and even, we have that F X T r is real and even, and F ∗ w l (0)

Original: =
Revised: =

Original: F w l (0) ’s are real and even.
Revised: F w l (0) ’s are real and even.

Original: By (66), we see that the time derivatives of F w l are also real and even.
Revised: By (66), we see that the time derivatives of F w l are also real and even.

Original: Thus, the parameters w l ( t ) are all real and even for all t ≥ 0 .
Revised: Thus, the parameters w l ( t ) are all real and even for all t ≥ 0 .

Original: From this observation, we can deﬁne η l ( t ) :
Revised: From this observation, we can deﬁne η l ( t ) :

Original: = F w l ( t ) , ¯ η :=
Revised: = F w l ( t ) , ¯ η :=

Original: F ¯ w , and S :
Revised: F ¯ w , and S :

Original: = d L − 1 2 Re( F ) , which are all real by the even symmetry.
Revised: = d L − 1 2 Re( F ) , which are all real by the even symmetry.

Original: Then, starting from (61), the proof goes through.
Revised: Then, starting from (61), the proof goes through.

Original: F. P ROOF OF C OROLLARY
Revised: F. P ROOF OF C OROLLARY

Original: Since the sensor matrices A 1 , . .
Revised: Since the sensor matrices A 1 , . .

Original: A n commute, they are simultaneously diagonalizable with a real unitary matrix U ∈ R d × d , i.e., U T A
Revised: A n commute, they are simultaneously diagonalizable with a real unitary matrix U ∈ R d × d , i.e., U T A

Original: i
Revised: i

Original: U ’s are diagonal matrices.
Revised: U ’s are diagonal matrices.

Original: From the deep matrix sensing problem (13), we can compute ∇ W l∂ L  , which gives the gradient ﬂow dynamics of W l .
Revised: From the deep matrix sensing problem (13), we can compute ∇ W l L ms , which gives the gradient ﬂow dynamics of W l .

Original: [Equation]
Revised: [Equation]

Original: where r i = (cid:104)
Revised: where r i = (cid:104)

Original: A i , W 1 · · · W L (cid:105) − y
Revised: A i , W 1 · · · W L (cid:105) − y

Original: i is the residual for the i -th sensor matrix.
Revised: i is the residual for the i -th sensor matrix.

Original: If all we left-multiply
Revised: If all we left-multiply

Original: U T and right-multiply U to both sides, we get
Revised: U T and right-multiply U to both sides, we get

Original: [Equation]
Revised: [Equation]

Original: If U T W
Revised: If U T W

Original: Tk U is a diagonal matrix for all k (cid:54) =
Revised: Tk U is a diagonal matrix for all k (cid:54) =

Original: l , then U T ˙
Revised: l , then U T ˙

Original: W l U is also a diagonal matrix.
Revised: W l U is also a diagonal matrix.

Original: Note also that, since W l (0)
Revised: Note also that, since W l (0)

Original: =
Revised: =

Original: α I d = α UU T for l ∈
Revised: α I d = α UU T for l ∈

Original: [ L − 1] , the product U T W l U is a diagonal matrix at initialization.
Revised: [ L − 1] , the product U T W l U is a diagonal matrix at initialization.

Original: These observations imply that W l ( t ) ’s are all diagonalizable with U for all t ≥ 0 .
Revised: These observations imply that W l ( t ) ’s are all diagonalizable with U for all t ≥ 0 .

Original: Now, deﬁne v l ( t ) =
Revised: Now, deﬁne v l ( t ) =

Original: eig( W l ( t )) , i.e., U T W l U = diag( v l ) .
Revised: eig( W l ( t )) , i.e., U T W l U = diag( v l ) .

Original: Also, let x i = eig
Revised: Also, let x i = eig

Original: ( A i ) .
Revised: ( A i ) .

Original: Then, (67) can be written as
Revised: Then, (67) can be written as

Original: [Equation]
Revised: [Equation]

Original: Therefore, this is equivalent to the regression problem with linear diagonal networks, initialized at v l (0) =
Revised: Therefore, this is equivalent to the regression problem with linear diagonal networks, initialized at v l (0) =

Original: α 1 for l ∈
Revised: α 1 for l ∈

Original: [ L − 1] and v L (0) = 0 .
Revised: [ L − 1] and v L (0) = 0 .

Original: Given this equivalence, Corollary 7 can be implied from
Revised: Given this equivalence, Corollary 7 can be implied from

Original: Corollary 5.
Revised: Corollary 5.

Original: G P ROOF OF T HEOREMG. C ONVERGENCE OF LOSS TO ZERO
Revised: G P ROOF OF T HEOREMG. C ONVERGENCE OF LOSS TO ZERO

Original: We ﬁrst show that given the conditions on initialization, the training loss L ( Θ ( t )) converges to zero.
Revised: We ﬁrst show that given the conditions on initialization, the training loss L ( Θ ( t )) converges to zero.

Original: Since L = 2 and M ( x ) =
Revised: Since L = 2 and M ( x ) =

Original: U 1 diag( s ) U T 2 , we can write the gradient ﬂow dynamics from Section 2.
Revised: U 1 diag( s ) U T 2 , we can write the gradient ﬂow dynamics from Section 2.

Original: as
Revised: as

Original: [Equation]
Revised: [Equation]

Original: where r ( t ) =
Revised: where r ( t ) =

Original: f ( x ; Θ ( t )) − y is the residual of the data point ( x , y ) .
Revised: f ( x ; Θ ( t )) − y is the residual of the data point ( x , y ) .

Original: From (68) we get
Revised: From (68) we get

Original: [Equation]
Revised: [Equation]

Original: Now consider the rate of growth for the j -th component of U T 1 v 1 squared:
Revised: Now consider the rate of growth for the j -th component of U T 1 v 1 squared:

Original: [Equation]
Revised: [Equation]

Original: So for any j ∈ [ m ]
Revised: So for any j ∈ [ m ]

Original: [Equation]
Revised: [Equation]

Original: [ U T 1 v 1 ] 2 j and [ U T 2 v 2 ] 2 j grow at the same rate.
Revised: [ U T 1 v 1 ] 2 j and [ U T 2 v 2 ] 2 j grow at the same rate.

Original: This means that the gap between the two layers stays constant for all t ≥ 0 .
Revised: This means that the gap between the two layers stays constant for all t ≥ 0 .

Original: Combining this with our conditions on initial directions,
Revised: Combining this with our conditions on initial directions,

Original: [Equation]
Revised: [Equation]

Original: for any j ∈ [ m ] and t ≥ 0 .
Revised: for any j ∈ [ m ] and t ≥ 0 .

Original: This inequality implies
Revised: This inequality implies

Original: [Equation]
Revised: [Equation]

Original: Let us now consider the time derivative of L ( Θ ( t )) .
Revised: Let us now consider the time derivative of L ( Θ ( t )) .

Original: We have the following chain of upper bounds on the time derivative:
Revised: We have the following chain of upper bounds on the time derivative:

Original: [Equation]
Revised: [Equation]

Original: where (a) used the fact that (cid:107) ˙ v 2 ( t ) (cid:107) 22 ≥ (cid:107)
Revised: where (a) used the fact that (cid:107) ˙ v 2 ( t ) (cid:107) 22 ≥ (cid:107)

Original: U 2 U T 2 ˙ v 2 ( t ) (cid:107) 22 because it is a projection onto a subspace, and (cid:107)
Revised: U 2 U T 2 ˙ v 2 ( t ) (cid:107) 22 because it is a projection onto a subspace, and (cid:107)

Original: U 2 U T 2 ˙ v L ( t ) (cid:107) 

Revised: U 2 U T 2 ˙ v L ( t ) (cid:107)

Original: = (cid:107) U T 2 ˙ v 2 ( t ) (cid:107) 

Revised: = (cid:107) U T 2 ˙ v 2 ( t ) (cid:107)

Original: because U T 2 U 2 = I k 2 ; (b) is due to (69); (c) is due to (70).
Revised: because U T 2 U 2 = I k 2 ; (b) is due to (69); (c) is due to (70).

Original: From this, we get
Revised: From this, we get

Original: [Equation]
Revised: [Equation]

Original: Therefore, L ( Θ ( t )) → 0 as t → ∞ .
Revised: Therefore, L ( Θ ( t )) → 0 as t → ∞ .

Original: G. C HARACTERIZING THE LIMIT POINT
Revised: G. C HARACTERIZING THE LIMIT POINT

Original: Now, we move on to characterize the limit points of the gradient ﬂow.
Revised: Now, we move on to characterize the limit points of the gradient ﬂow.

Original: First, note that any changes made in v l over time are in the subspace spanned by the columns of U l .
Revised: First, note that any changes made in v l over time are in the subspace spanned by the columns of U l .

Original: Therefore, any component in the initialization v l (0) =
Revised: Therefore, any component in the initialization v l (0) =

Original: α ¯ v l that is orthogonal to the column space of U l stays constant.
Revised: α ¯ v l that is orthogonal to the column space of U l stays constant.

Original: So, we can focus on the evolution of v l in the column space of U l ; this can be done by deﬁning a “transformed” version of the parameters η l ( t ) :
Revised: So, we can focus on the evolution of v l in the column space of U l ; this can be done by deﬁning a “transformed” version of the parameters η l ( t ) :

Original: = U Tl v l ( t ) and using (69), one can deﬁne an equivalent system of ODEs:
Revised: = U Tl v l ( t ) and using (69), one can deﬁne an equivalent system of ODEs:

Original: [Equation]
Revised: [Equation]

Original: where ¯ η 1 :
Revised: where ¯ η 1 :

Original: = U T 1 ¯ v 1 , ¯ η 2 :=
Revised: = U T 1 ¯ v 1 , ¯ η 2 :=

Original: U T 2 ¯ v 2 .
Revised: U T 2 ¯ v 2 .

Original: It is straightforward to verify that the solution to (72) has the following form.
Revised: It is straightforward to verify that the solution to (72) has the following form.

Original: [Equation]
Revised: [Equation]

Original: By the convergence of the loss to zero (71)
Revised: By the convergence of the loss to zero (71)

Original: [Equation]
Revised: [Equation]

Original: we have lim t →∞ f ( x ; Θ ( t ))
Revised: we have lim t →∞ f ( x ; Θ ( t ))

Original: =
Revised: =

Original: y .
Revised: y .

Original: Note that f ( x ; Θ ( t )) can be written as
Revised: Note that f ( x ; Θ ( t )) can be written as

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: where we deﬁned ν := − (cid:82) ∞ 0 r ( τ ) dτ .
Revised: where we deﬁned ν := − (cid:82) ∞ 0 r ( τ ) dτ .

Original: Consider the function ν
Revised: Consider the function ν

Original: (cid:55)→ a sinh( ν ) + b cosh( ν ) .
Revised: (cid:55)→ a sinh( ν ) + b cosh( ν ) .

Original: This is a strictly increasing function if a > | b | .
Revised: This is a strictly increasing function if a > | b | .

Original: Note also that
Revised: Note also that

Original: [Equation]
Revised: [Equation]

Original: 75) which holds with equality if and only if | [¯ η 1 ] j | = | [¯ η 2 ] j | .
Revised: 75) which holds with equality if and only if | [¯ η 1 ] j | = | [¯ η 2 ] j | .

Original: However, recall from our assumptions on initialization that [¯ η 1 ] 2 j − [¯ η 2 ] 2 j ≥ λ > 0 , so (75) can only hold with strict inequality.
Revised: However, recall from our assumptions on initialization that [¯ η 1 ] 2 j − [¯ η 2 ] 2 j ≥ λ > 0 , so (75) can only hold with strict inequality.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: is a strictly increasing (hence invertible) function because it is a sum of m strictly increasing function.
Revised: is a strictly increasing (hence invertible) function because it is a sum of m strictly increasing function.

Original: Using this g ( ν ) , (74) can be written as α 2 g ( ν ) =
Revised: Using this g ( ν ) , (74) can be written as α 2 g ( ν ) =

Original: y , and by using the inverse of g
Revised: y , and by using the inverse of g

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Plugging (76) into (73), we get
Revised: Plugging (76) into (73), we get

Original: [Equation]
Revised: [Equation]

Original: This ﬁnishes the proof.
Revised: This ﬁnishes the proof.

Original: H P ROOF OF T HEOREMH. C ONVERGENCE OF LOSS TO ZERO
Revised: H P ROOF OF T HEOREMH. C ONVERGENCE OF LOSS TO ZERO

Original: We ﬁrst show that given the conditions on initialization, the training loss L ( Θ ( t )) converges to zero.
Revised: We ﬁrst show that given the conditions on initialization, the training loss L ( Θ ( t )) converges to zero.

Original: Recall from (10) that the linear fully-connected network can be written as
Revised: Recall from (10) that the linear fully-connected network can be written as

Original: [Equation]
Revised: [Equation]

Original: From the deﬁnition of the training loss L
Revised: From the deﬁnition of the training loss L

Original: [Equation]
Revised: [Equation]

Original: it is straightforward to check that the gradient ﬂow dynamics read
Revised: it is straightforward to check that the gradient ﬂow dynamics read

Original: [Equation]
Revised: [Equation]

Original: where r ∈ R n is the residual vector satisfying [ r ] i
Revised: where r ∈ R n is the residual vector satisfying [ r ] i

Original: = f fc ( x i ; Θ fc ) − y i , as deﬁned in Section 2.1.
Revised: = f fc ( x i ; Θ fc ) − y i , as deﬁned in Section 2.1.

Original: From (77)
Revised: From (77)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: for any l ∈ [ L − 2] .
Revised: for any l ∈ [ L − 2] .

Original: From this, we have
Revised: From this, we have

Original: [Equation]
Revised: [Equation]

Original: and thus
Revised: and thus

Original: [Equation]
Revised: [Equation]

Original: for any l ∈ [ L − 2] .
Revised: for any l ∈ [ L − 2] .

Original: Similarly, we have
Revised: Similarly, we have

Original: [Equation]
Revised: [Equation]

Original: Let us now consider the time derivative of L ( Θ fc ( t )) .
Revised: Let us now consider the time derivative of L ( Θ fc ( t )) .

Original: We have the following chain of upper bounds on the time derivative:
Revised: We have the following chain of upper bounds on the time derivative:

Original: [Equation]
Revised: [Equation]

Original: Note from (80) that if W TL − 1 · · · W T 1 is full-rank, its minimum singular value is positive
Revised: Note from (80) that if W TL − 1 · · · W T 1 is full-rank, its minimum singular value is positive

Original: [Equation]
Revised: [Equation]

Original: and one can bound
Revised: and one can bound

Original: [Equation]
Revised: [Equation]

Original: We now prove that the matrix W TL − 1 · · · W T 1 is full-rank
Revised: We now prove that the matrix W TL − 1 · · · W T 1 is full-rank

Original: [Equation]
Revised: [Equation]

Original: and its minimum singular value is bounded from below by α L − 1 λ ( L − 1) / 2 for any t ≥ 0 .
Revised: and its minimum singular value is bounded from below by α L − 1 λ ( L − 1) / 2 for any t ≥ 0 .

Original: To show this, it sufﬁces to show that
Revised: To show this, it sufﬁces to show that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now,
Revised: Now,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where equalities marked in (a) used (78), and inequalities marked in (b) used the initialization conditions ¯ W Tl ¯ W l (cid:23) ¯ W l +1 ¯ W Tl +1 .
Revised: where equalities marked in (a) used (78), and inequalities marked in (b) used the initialization conditions ¯ W Tl ¯ W l (cid:23) ¯ W l +1 ¯ W Tl +1 .

Original: Next, it follows from (79) that
Revised: Next, it follows from (79) that

Original: [Equation]
Revised: [Equation]

Original: where (c) used the assumption that ¯ W TL − 1 ¯ W L − 1 − ¯ w L ¯ w TL (cid:23) λ I d .
Revised: where (c) used the assumption that ¯ W TL − 1 ¯ W L − 1 − ¯ w L ¯ w TL (cid:23) λ I d .

Original: This proves (82).
Revised: This proves (82).

Original: Applying (82) to (80) then gives
Revised: Applying (82) to (80) then gives

Original: [Equation]
Revised: [Equation]

Original: where (d) used the fact that X T is a full column rank matrix to apply a bound similar to (81).
Revised: where (d) used the fact that X T is a full column rank matrix to apply a bound similar to (81).

Original: From this, we get
Revised: From this, we get

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: , hence proving L ( Θ fc ( t ))
Revised: , hence proving L ( Θ fc ( t ))

Original: → 0 as t → ∞ .
Revised: → 0 as t → ∞ .

Original: H. C HARACTERIZING THE LIMIT POINT : α → 0 CASE
Revised: H. C HARACTERIZING THE LIMIT POINT : α → 0 CASE

Original: Now, we move on to characterize the limit points of the gradient ﬂow, for the “active regime” case α → 0 .
Revised: Now, we move on to characterize the limit points of the gradient ﬂow, for the “active regime” case α → 0 .