Original: This part of the proof is motivated from the analysis in Ji & Telgarsky (2019a).
Revised: This part of the proof is motivated from the analysis in Ji & Telgarsky (2019a).

Original: Let u l and v l be the top left and right singular vectors of W l , for l ∈
Revised: Let u l and v l be the top left and right singular vectors of W l , for l ∈

Original: [ L − 1] .
Revised: [ L − 1] .

Original: Note that since W l varies over time, the singular vectors and singular value also vary over time.
Revised: Note that since W l varies over time, the singular vectors and singular value also vary over time.

Original: Similarly, let s l be the largest singular value of W l .
Revised: Similarly, let s l be the largest singular value of W l .

Original: We will show that the linear coefﬁcients β fc ( Θ fc ) =
Revised: We will show that the linear coefﬁcients β fc ( Θ fc ) =

Original: W 1 · · · W L − 1 w L align with u 1 as α → 0 , and u 1 is in the subspace of row( X ) in the limit α → 0 , hence proving that β fc ( Θ fc ) is the minimum (cid:96) 2 norm solution in the limit α → 0 .
Revised: W 1 · · · W L − 1 w L align with u 1 as α → 0 , and u 1 is in the subspace of row( X ) in the limit α → 0 , hence proving that β fc ( Θ fc ) is the minimum (cid:96) 2 norm solution in the limit α → 0 .

Original: First, note from (78) and (79) that if we take trace of both sides, we get
Revised: First, note from (78) and (79) that if we take trace of both sides, we get

Original: [Equation]
Revised: [Equation]

Original: Summing the equations above for l, l + 1 , . .
Revised: Summing the equations above for l, l + 1 , . .

Original: , L − 1 , we get
Revised: , L − 1 , we get

Original: [Equation]
Revised: [Equation]

Original: Next, consider the operator norms (i.e., the maximum singular values), denoted as (cid:107)·(cid:107) 2 , of the matrices.
Revised: Next, consider the operator norms (i.e., the maximum singular values), denoted as (cid:107)·(cid:107) 2 , of the matrices.

Original: [Equation]
Revised: [Equation]

Original: Tl
Revised: Tl

Original: [Equation]
Revised: [Equation]

Original: for l ∈
Revised: for l ∈

Original: [ L − 2]
Revised: [ L − 2]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (e) used (78) and (f) used (79).
Revised: where (e) used (78) and (f) used (79).

Original: Summing the inequalities gives
Revised: Summing the inequalities gives

Original: [Equation]
Revised: [Equation]

Original: From (83) and (84)
Revised: From (83) and (84)

Original: [Equation]
Revised: [Equation]

Original: we get a bound on the gap between the second powers of the Frobenius norm (or the (cid:96) 2 norm of singular values) and operator norm (or the maximum singular value s l ) of W l :
Revised: we get a bound on the gap between the second powers of the Frobenius norm (or the (cid:96) 2 norm of singular values) and operator norm (or the maximum singular value s l ) of W l :

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: which holds for any t ≥ 0 .
Revised: which holds for any t ≥ 0 .

Original: The gap (85) implies that each W l , for l ∈ [ L − 1] , can be written as
Revised: The gap (85) implies that each W l , for l ∈ [ L − 1] , can be written as

Original: [Equation]
Revised: [Equation]

Original: Next, we show that the “adjacent” singular vectors v l and u l +1 align with each other as α → 0 .
Revised: Next, we show that the “adjacent” singular vectors v l and u l +1 align with each other as α → 0 .

Original: To this end, we will get lower and upper bounds for a quantity v Tl W l +1 W Tl +1 v l .
Revised: To this end, we will get lower and upper bounds for a quantity v Tl W l +1 W Tl +1 v l .

Original: [Equation]
Revised: [Equation]

Original: Combining (87), (88)
Revised: Combining (87), (88)

Original: [Equation]
Revised: [Equation]

Original: and (85)
Revised: and (85)

Original: [Equation]
Revised: [Equation]

Original: we get s
Revised: we get s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Next, by a similar reasoning as (87)
Revised: Next, by a similar reasoning as (87)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Combining (89) and (90) and dividing both sides by s 2 l +
Revised: Combining (89) and (90) and dividing both sides by s 2 l +

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for t ≥
Revised: for t ≥

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: By a similar argument
Revised: By a similar argument

Original: [Equation]
Revised: [Equation]

Original: we can also get
Revised: we can also get

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: From (91) and (92), we can note that as α → 0 , the inner product between the adjacent singular vectors converges to ± 1 , unless s 2 , . . .
Revised: From (91) and (92), we can note that as α → 0 , the inner product between the adjacent singular vectors converges to ± 1 , unless s 2 , . . .

Original: , s L − 1 , (cid:107) w L (cid:107) 2 also diminish to zero.
Revised: , s L − 1 , (cid:107) w L (cid:107) 2 also diminish to zero.

Original: So it is left to show that the singular values do not diminish to zero as α → 0 .
Revised: So it is left to show that the singular values do not diminish to zero as α → 0 .

Original: To this end, recall that we proved in the previous subsection that
Revised: To this end, recall that we proved in the previous subsection that

Original: [Equation]
Revised: [Equation]

Original: A necessary condition for this to hold is that
Revised: A necessary condition for this to hold is that

Original: [Equation]
Revised: [Equation]

Original: This means that after converging to the global minimum solution of the problem (i
Revised: This means that after converging to the global minimum solution of the problem (i

Original: [Equation]
Revised: [Equation]

Original: e., t → ∞ ), the product of the singular values must be at least greater than some constant independent of α .
Revised: e., t → ∞ ), the product of the singular values must be at least greater than some constant independent of α .

Original: Moreover, we can see from (87) and (90) that the gap between singular values squared of adjacent layers is bounded by O ( α 2 ) , for all t ≥ 0 ; so the maximum singular values become closer and closer to each other as α diminishes.
Revised: Moreover, we can see from (87) and (90) that the gap between singular values squared of adjacent layers is bounded by O ( α 2 ) , for all t ≥ 0 ; so the maximum singular values become closer and closer to each other as α diminishes.

Original: This implies that
Revised: This implies that

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have the alignment of singular vectors at convergence as α → 0 :
Revised: we have the alignment of singular vectors at convergence as α → 0 :

Original: [Equation]
Revised: [Equation]

Original: So far, we saw from (86) that W l ( t ) ’s become rank-1 matrices as α → 0 , and from (93) that the top singular vectors align with each other as t → ∞ and α → 0 .
Revised: So far, we saw from (86) that W l ( t ) ’s become rank-1 matrices as α → 0 , and from (93) that the top singular vectors align with each other as t → ∞ and α → 0 .

Original: These imply that, as t → ∞ and α → 0 , β fc ( Θ fc ) is a scalar multiple of the u 1 , the top left singular vector of W 1 :
Revised: These imply that, as t → ∞ and α → 0 , β fc ( Θ fc ) is a scalar multiple of the u 1 , the top left singular vector of W 1 :

Original: [Equation]
Revised: [Equation]

Original: for some c ∈ R .
Revised: for some c ∈ R .

Original: In light of (94), it remains to take a close look at u 1 ( t ) .
Revised: In light of (94), it remains to take a close look at u 1 ( t ) .

Original: Note from the gradient ﬂow dynamics of Wthat ˙ W 1 is always a rank-1 matrix whose columns are in the row space of X , since X T r ∈ row( X ) .
Revised: Note from the gradient ﬂow dynamics of Wthat ˙ W 1 is always a rank-1 matrix whose columns are in the row space of X , since X T r ∈ row( X ) .

Original: This implies that, if we decompose W 1 into two orthogonal components W ⊥ 1 and W (cid:107) 1 so that the columns in W (cid:107) 1 are in row( X ) and the columns in W ⊥ 1 are in the orthogonal subspace row( X ) ⊥ , we have
Revised: This implies that, if we decompose W 1 into two orthogonal components W ⊥ 1 and W (cid:107) 1 so that the columns in W (cid:107) 1 are in row( X ) and the columns in W ⊥ 1 are in the orthogonal subspace row( X ) ⊥ , we have

Original: [Equation]
Revised: [Equation]

Original: That is, any component W ⊥ 1 (0) orthogonal to row( X ) remains unchanged for all t ≥ 0 , while the component W (cid:107) 1 changes by the gradient ﬂow.
Revised: That is, any component W ⊥ 1 (0) orthogonal to row( X ) remains unchanged for all t ≥ 0 , while the component W (cid:107) 1 changes by the gradient ﬂow.

Original: Since we have
Revised: Since we have

Original: [Equation]
Revised: [Equation]

Original: , the component in W 1 that is orthogonal to row( X ) diminishes to zero as α → 0 .
Revised: , the component in W 1 that is orthogonal to row( X ) diminishes to zero as α → 0 .

Original: This means that at the limit α → 0 , the columns of W 1 are entirely from row( X ) , which also means that
Revised: This means that at the limit α → 0 , the columns of W 1 are entirely from row( X ) , which also means that

Original: [Equation]
Revised: [Equation]

Original: However, recall that there is only one unique global minimum of Xz = y in row( X ) : namely, z = X T ( XX T ) − 1 y , the minimum (cid:96) 2 norm solution.
Revised: However, recall that there is only one unique global minimum of Xz = y in row( X ) : namely, z = X T ( XX T ) − 1 y , the minimum (cid:96) 2 norm solution.

Original: This ﬁnishes the proof.
Revised: This ﬁnishes the proof.