Original: T HE I MPLICIT B IAS OF D EPTH : H OW I NCREMENTAL L EARNING D RIVES G ENERALIZATION
Revised: T HE I MPLICIT B IAS OF D EPTH : H OW I NCREMENTAL L EARNING D RIVES G ENERALIZATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity.
Revised: A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity.

Original: We formally deﬁne the notion of incremental learning dynamics and derive the conditions on depth and initialization for which this phenomenon arises in deep linear models.
Revised: We formally deﬁne the notion of incremental learning dynamics and derive the conditions on depth and initialization for which this phenomenon arises in deep linear models.

Original: Our main theoretical contribution is a dynamical depth separation result, proving that while shallow models can exhibit incremental learning dynamics, they require the initialization to be exponentially small for these dynamics to present themselves.
Revised: Our main theoretical contribution is a dynamical depth separation result, proving that while shallow models can exhibit incremental learning dynamics, they require the initialization to be exponentially small for these dynamics to present themselves.

Original: However, once the model becomes deeper, the dependence becomes polynomial and incremental learning can arise in more natural settings.
Revised: However, once the model becomes deeper, the dependence becomes polynomial and incremental learning can arise in more natural settings.

Original: We complement our theoretical ﬁndings by experimenting with deep matrix sensing, quadratic neural networks and with binary classiﬁcation using diagonal and convolutional linear networks, showing all of these models exhibit incremental learning.
Revised: We complement our theoretical ﬁndings by experimenting with deep matrix sensing, quadratic neural networks and with binary classiﬁcation using diagonal and convolutional linear networks, showing all of these models exhibit incremental learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks have led to a breakthrough in modern machine learning, allowing us to efﬁciently learn highly expressive models that still generalize to unseen data.
Revised: Neural networks have led to a breakthrough in modern machine learning, allowing us to efﬁciently learn highly expressive models that still generalize to unseen data.

Original: The theoretical reasons for this success are still unclear, as the generalization capabilities of neural networks defy the classic statistical learning theory bounds.
Revised: The theoretical reasons for this success are still unclear, as the generalization capabilities of neural networks defy the classic statistical learning theory bounds.

Original: Since these bounds, which depend solely on the capacity of the learned model, are unable to account for the success of neural networks, we must examine additional properties of the learning process.
Revised: Since these bounds, which depend solely on the capacity of the learned model, are unable to account for the success of neural networks, we must examine additional properties of the learning process.

Original: One such property is the optimization algorithm - while neural networks can express a multitude of possible ERM solutions for a given training set, gradient-based methods with the right initialization may be implicitly biased towards certain solutions which generalize.
Revised: One such property is the optimization algorithm - while neural networks can express a multitude of possible ERM solutions for a given training set, gradient-based methods with the right initialization may be implicitly biased towards certain solutions which generalize.

Original: A possible way such an implicit bias may present itself, is if gradient-based methods were to search the hypothesis space for possible solutions of gradually increasing complexity.
Revised: A possible way such an implicit bias may present itself, is if gradient-based methods were to search the hypothesis space for possible solutions of gradually increasing complexity.

Original: This would suggest that while the hypothesis space itself is extremely complex, our search strategy favors the simplest solutions and thus generalizes.
Revised: This would suggest that while the hypothesis space itself is extremely complex, our search strategy favors the simplest solutions and thus generalizes.

Original: One of the leading results along these lines has been by Saxe et al.
Revised: One of the leading results along these lines has been by Saxe et al.

Original: (2013), deriving an analytical solution for the gradient ﬂow dynamics of deep linear networks and showing that for such models, the singular values converge at different rates, with larger values converging ﬁrst.
Revised: (2013), deriving an analytical solution for the gradient ﬂow dynamics of deep linear networks and showing that for such models, the singular values converge at different rates, with larger values converging ﬁrst.

Original: At the limit of inﬁnitesimal initialization of the deep linear network, Gidel et al.
Revised: At the limit of inﬁnitesimal initialization of the deep linear network, Gidel et al.

Original: (2019) show these dynamics exhibit a behavior of “incremental learning” - the singular values of the model are learned separately, one at a time.
Revised: (2019) show these dynamics exhibit a behavior of “incremental learning” - the singular values of the model are learned separately, one at a time.

Original: Our work generalizes these results to small but ﬁnite initialization scales.
Revised: Our work generalizes these results to small but ﬁnite initialization scales.

Original: Incremental learning dynamics have also been explored in gradient descent applied to matrix completion and sensing with a factorized parameterization (Gunasekar et al.
Revised: Incremental learning dynamics have also been explored in gradient descent applied to matrix completion and sensing with a factorized parameterization (Gunasekar et al.

Original: (2017), Arora et al.
Revised: (2017), Arora et al.

Original: Woodworth et al.
Revised: Woodworth et al.

Original: When initialized with small Gaussian weights and trained with a small learning rate, such a model is able to successfully recover the low-rank matrix which labeled the data, even if the problem is highly over-determined and no additional regularization is applied.
Revised: When initialized with small Gaussian weights and trained with a small learning rate, such a model is able to successfully recover the low-rank matrix which labeled the data, even if the problem is highly over-determined and no additional regularization is applied.

Original: In their proof of low-rank recovery for such models, Li et al.
Revised: In their proof of low-rank recovery for such models, Li et al.

Original: (2017) show that the model remains lowrank throughout the optimization process, leading to the successful generalization.
Revised: (2017) show that the model remains lowrank throughout the optimization process, leading to the successful generalization.

Original: Additionally,
Revised: Additionally,

Revised: [Figure]

Original: Arora et al.
Revised: Arora et al.

Original: (2019) explore the dynamics of such models, showing the singular values are learned at different rates and that deeper models exhibit stronger incremental learning dynamics.
Revised: (2019) explore the dynamics of such models, showing the singular values are learned at different rates and that deeper models exhibit stronger incremental learning dynamics.

Original: Our work[Figure] deals with a more simpliﬁed setting, allowing us to determine explicitly under which conditions depth leads to this dynamical phenomenon.
Revised: Our work deals with a more simpliﬁed setting, allowing us to determine explicitly under which conditions depth leads to this dynamical phenomenon.

Original: Finally, the learning dynamics of nonlinear models have been studied as well.
Revised: Finally, the learning dynamics of nonlinear models have been studied as well.

Original: Combes et al.
Revised: Combes et al.

Original: (2018) and Williams et al.
Revised: (2018) and Williams et al.

Original: (2019) study the gradient ﬂow dynamics of shallow ReLU networks under restrictive distributional assumptions, Basri et al.
Revised: (2019) study the gradient ﬂow dynamics of shallow ReLU networks under restrictive distributional assumptions, Ronen et al.

Original: (2019) show that shallow networks learn functions of gradually increasing frequencies and Nakkiran et al.
Revised: (2019) show that shallow networks learn functions of gradually increasing frequencies and Nakkiran et al.

Original: (2019) show how deep ReLU networks correlate with linear classiﬁers in the early stages of training.
Revised: (2019) show how deep ReLU networks correlate with linear classiﬁers in the early stages of training.

Original: These ﬁndings, along with others, suggest that the generalization ability of deep networks is at least in part due to the incremental learning dynamics of gradient descent.
Revised: These ﬁndings, along with others, suggest that the generalization ability of deep networks is at least in part due to the incremental learning dynamics of gradient descent.

Original: Following this line of work, we begin by explicitly deﬁning the notion of incremental learning for a toy model which exhibits this sort of behavior.
Revised: Following this line of work, we begin by explicitly deﬁning the notion of incremental learning for a toy model which exhibits this sort of behavior.

Original: Analyzing the dynamics of the model for gradient ﬂow and gradient descent, we characterize the effect of the model’s depth and initialization scale on incremental learning, showing how deeper models allow for incremental learning in larger (realistic) initialization scales.
Revised: Analyzing the dynamics of the model for gradient ﬂow and gradient descent, we characterize the effect of the model’s depth and initialization scale on incremental learning, showing how deeper models allow for incremental learning in larger (realistic) initialization scales.

Original: Speciﬁcally, we show that a depth-2 model requires exponentially small initialization for incremental learning to occur, while deeper models only require the initialization to be polynomially small.
Revised: Speciﬁcally, we show that a depth-2 model requires exponentially small initialization for incremental learning to occur, while deeper models only require the initialization to be polynomially small.

Original: Once incremental learning has been deﬁned and characterized for the toy model, we generalize our results theoretically and empirically for larger linear and quadratic models.
Revised: Once incremental learning has been deﬁned and characterized for the toy model, we generalize our results theoretically and empirically for larger linear and quadratic models.

Original: Examples of incremental learning in these models can be seen in ﬁgure 1, which we discuss further in section 4.
Revised: Examples of incremental learning in these models can be seen in ﬁgure 1, which we discuss further in section 4.

Original:  D YNAMICAL A NALYSIS OF A T OY M ODEL
Revised:  D YNAMICAL A NALYSIS OF A T OY M ODEL

Original: We begin by analyzing incremental learning for a simple model.
Revised: We begin by analyzing incremental learning for a simple model.

Original: This will allow us to gain a clear understanding of the phenomenon and the conditions for it, which we will later be able to apply to a variety of other models in which incremental learning is present.
Revised: This will allow us to gain a clear understanding of the phenomenon and the conditions for it, which we will later be able to apply to a variety of other models in which incremental learning is present.

Original: 2. P RELIMINARIES
Revised: 2. P RELIMINARIES

Original: Our simple linear model will be similar to the toy model analyzed by Woodworth et al.
Revised: Our simple linear model will be similar to the toy model analyzed by Woodworth et al.

Original: Our input space will be X
Revised: Our input space will be X

Original: = R d and the hypothesis space will be linear models with non-negative weights, such that:
Revised: = R d and the hypothesis space will be linear models with non-negative weights, such that:

Original: [Equation]
Revised: [Equation]

Original: We will introduce depth into our model, by parameterizing σ using w ∈ R d ≥ 0 in the following way:
Revised: We will introduce depth into our model, by parameterizing σ using w ∈ R d ≥ 0 in the following way:

Original: [Equation]
Revised: [Equation]

Original: Where N represents the depth of the model.
Revised: Where N represents the depth of the model.

Original: Since we restrict the model to having non-negative weights, this parameterization doesn’t change the expressiveness, but it does radically change it’s optimization dynamics.
Revised: Since we restrict the model to having non-negative weights, this parameterization doesn’t change the expressiveness, but it does radically change it’s optimization dynamics.

Original: Assuming the data is labeled by some σ ∗ ∈ R d ≥ 0 , we will study the dynamics of this model for general N under a depth-normalized 1 squared loss over Gaussian inputs, which will allow us to derive our analytical solution:
Revised: Assuming the data is labeled by some σ ∗ ∈ R d ≥ 0 , we will study the dynamics of this model for general N under a depth-normalized 1 squared loss over Gaussian inputs, which will allow us to derive our analytical solution:

Original: [Equation]
Revised: [Equation]

Original: We will assume that our model is initialized uniformly with a tunable scaling factor, such that:
Revised: We will assume that our model is initialized uniformly with a tunable scaling factor, such that:

Original: [Equation]
Revised: [Equation]

Original: 2. G RADIENT F LOW A NALYTICAL S OLUTIONS
Revised: 2. G RADIENT F LOW A NALYTICAL S OLUTIONS

Original: Analyzing our toy model using gradient ﬂow allows us to obtain an analytical solution for the dynamics of σ ( t ) along with the dynamics of the loss function for a general N .
Revised: Analyzing our toy model using gradient ﬂow allows us to obtain an analytical solution for the dynamics of σ ( t ) along with the dynamics of the loss function for a general N .

Original: For brevity, the following theorem refers only to N = 1 , 2 and N → ∞ , however the solutions for 3 ≤ N < ∞ are similar in structure to N → ∞ , but more complicated.
Revised: For brevity, the following theorem refers only to N = 1 , 2 and N → ∞ , however the solutions for 3 ≤ N < ∞ are similar in structure to N → ∞ , but more complicated.

Original: We also assume σ ∗ i > 0 for brevity, however we can derive the solutions for σ ∗ i = 0 as well.
Revised: We also assume σ ∗ i > 0 for brevity, however we can derive the solutions for σ ∗ i = 0 as well.

Original: Note that this result is a special case adaptation of the one presented in Saxe et al.
Revised: Note that this result is a special case adaptation of the one presented in Saxe et al.

Original: (2013) for deep linear networks:
Revised: (2013) for deep linear networks:

Original: Theorem 1.
Revised: Theorem 1.

Original: Minimizing the toy linear model described in equation 1 with gradient ﬂow over the depth normalized squared loss equation 3, with Gaussian inputs and weights initialized as in equation 4 and assuming σ ∗ i > 0 leads to the following analytical solutions for different values of N :
Revised: Minimizing the toy linear model described in (1) with gradient ﬂow over the depth normalized squared loss (2) , with Gaussian inputs and weights initialized as in (3) and assuming σ ∗ i > 0 leads to the following analytical solutions for different values of N :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The gradient ﬂow equations for our model are the following:
Revised: The gradient ﬂow equations for our model are the following:

Original: [Equation]
Revised: [Equation]

Original: Given the dynamics of the w parameters, we may use the chain rule to derive the dynamics of the induced model, σ :
Revised: Given the dynamics of the w parameters, we may use the chain rule to derive the dynamics of the induced model, σ :

Original: [Equation]
Revised: [Equation]

Original: This differential equation is solvable for all N , leading to the solutions in the theorem.
Revised: This differential equation is solvable for all N , leading to the solutions in the theorem.

Original: Taking
Revised: Taking

Original: N → ∞ in equation 8 leads to ˙ 
Revised: N → ∞ in (4) leads to ˙ σ i

Original: σ i =
Revised: =

Original: σ 2 i ( σ ∗ i − σ i ) , which is also solvable.
Revised: σ 2 i ( σ ∗ i − σ i ) , which is also solvable.

Original: Analyzing these solutions, we see how even in such a simple model depth causes different factors of the model to be learned at different rates.
Revised: Analyzing these solutions, we see how even in such a simple model depth causes different factors of the model to be learned at different rates.

Original: Speciﬁcally, values corresponding to larger optimal values converge faster, suggesting a form of incremental learning.
Revised: Speciﬁcally, values corresponding to larger optimal values converge faster, suggesting a form of incremental learning.

Original: This is most clear for N = 2 where the solution isn’t implicit, but is also the case for N ≥ 3 , as we will see in the next subsection.
Revised: This is most clear for N = 2 where the solution isn’t implicit, but is also the case for N ≥ 3 , as we will see in the next subsection.

Original: These dynamics are depicted in ﬁgure 2, where we see the dynamics of the different values of σ ( t ) as learning progresses.
Revised: These dynamics are depicted in ﬁgure 2, where we see the dynamics of the different values of σ ( t ) as learning progresses.

Original: When N = 1 , all values are learned at the same rate regardless of the initialization, while the deeper models are clearly biased towards learning the larger singular values ﬁrst, especially at small initialization scales.
Revised: When N = 1 , all values are learned at the same rate regardless of the initialization, while the deeper models are clearly biased towards learning the larger singular values ﬁrst, especially at small initialization scales.

Original: Our model has only one optimal solution due to the population loss, but it is clear how this sort of dynamic can induce sparse solutions - if the model is able to ﬁt the data after a small amount of learning phases, then it’s obtained result will be sparse.
Revised: Our model has only one optimal solution due to the population loss, but it is clear how this sort of dynamic can induce sparse solutions - if the model is able to ﬁt the data after a small amount of learning phases, then it’s obtained result will be sparse.

Original: Alternatively, if N = 1 , we know that the dynamics will lead to the minimal (cid:96) 2 norm solution which is dense.
Revised: Alternatively, if N = 1 , we know that the dynamics will lead to the minimal (cid:96) 2 norm solution which is dense.

Original: We explore the sparsity inducing bias of our toy model by comparing it empirically 2 to a greedy sparse approximation algorithm in appendix D, and give our theoretical results in the next section.
Revised: We explore the sparsity inducing bias of our toy model by comparing it empirically 2 to a greedy sparse approximation algorithm in appendix D, and give our theoretical results in the next section.

Original:  I NCREMENTAL L EARNING
Revised:  I NCREMENTAL L EARNING

Original: Equipped with analytical solutions for the dynamics of our model for every depth, we turn to study how the depth and initialization effect incremental learning.
Revised: Equipped with analytical solutions for the dynamics of our model for every depth, we turn to study how the depth and initialization effect incremental learning.

Original: While Gidel et al.
Revised: While Gidel et al.

Original: (2019) focuses on incremental learning in depth-2 models at the limit of σ 0 → 0 , we will study the phenomenon for a general depth and for σ 0 > 0 .
Revised: (2019) focuses on incremental learning in depth-2 models at the limit of σ 0 → 0 , we will study the phenomenon for a general depth and for σ 0 > 0 .

Original: First, we will deﬁne the notion of incremental learning.
Revised: First, we will deﬁne the notion of incremental learning.

Original: Since all values of σ are learned in parallel, we can’t expect one value to converge before the other moves at all (which happens for inﬁnitesimal initialization as shown by Gidel et al.
Revised: Since all values of σ are learned in parallel, we can’t expect one value to converge before the other moves at all (which happens for inﬁnitesimal initialization as shown by Gidel et al.

Original: We will need a more relaxed deﬁnition for incremental learning in ﬁnite initialization scales.
Revised: We will need a more relaxed deﬁnition for incremental learning in ﬁnite initialization scales.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Given two values σ i , σ j such that σ ∗ i > σ ∗ j > 0 and both are initialized as σ i (0) =
Revised: Given two values σ i , σ j such that σ ∗ i > σ ∗ j > 0 and both are initialized as σ i (0) =

Original: σ j (0) = σ 0 < σ ∗ j , and given two scalars s ∈ (0 , 14 ) and f ∈ ( 34 , 1) , we call the learning of the values ( s, f ) -incremental
Revised: σ j (0) = σ 0 < σ ∗ j , and given two scalars s ∈ (0 , 14 ) and f ∈ ( 34 , 1) , we call the learning of the values ( s, f ) -incremental

Original: if there exists a t for which:
Revised: if there exists a t for which:

Original: [Equation]
Revised: [Equation]

Original: In words, two values have distinct learning phases if the ﬁrst almost converges ( f ≈ 1 ) before the second changes by much ( s (cid:28) 1 ).
Revised: In words, two values have distinct learning phases if the ﬁrst almost converges ( f ≈ 1 ) before the second changes by much ( s (cid:28) 1 ).

Original: Note that for any N , σ ( t ) is monotonically increasing and so once σ j ( t )
Revised: Note that for any N , σ ( t ) is monotonically increasing and so once σ j ( t )

Original: =
Revised: =

Original: sσ ∗ j , it will not decrease to allow further incremental learning.
Revised: sσ ∗ j , it will not decrease to allow further incremental learning.

Original: Given this deﬁnition of incremental learning, we turn to study the conditions that facilitate incremental learning in our toy model.
Revised: Given this deﬁnition of incremental learning, we turn to study the conditions that facilitate incremental learning in our toy model.

Original: Our main result is a dynamical depth separation result, showing that incremental learning is dependent on σ ∗ i σ ∗ j in different ways for different values of N .
Revised: Our main result is a dynamical depth separation result, showing that incremental learning is dependent on σ ∗ i σ ∗ j in different ways for different values of N .

Original: The largest difference in dependence happens between N = 2 and N = 3 , where the dependence changes from exponential to polynomial:
Revised: The largest difference in dependence happens between N = 2 and N = 3 , where the dependence changes from exponential to polynomial:

Original: Theorem 2.
Revised: Theorem 2.

Original: Given two values σ i , σ j of a toy linear model as in equation 1, where σ ∗ i σ ∗ j = r >and the model is initialized as in equation 4, and given two scalars s ∈ (0 , 1 4 ) and f ∈ ( 34 , 1) , then the largest initialization value for which the learning phases of the values are ( s, f ) -incremental, denoted σ th 0 , is bounded in the following way:
Revised: Given two values σ i , σ j of a toy linear model as in (1) , where σ ∗ i σ ∗ j = r > 1 and the model is initialized as in (3) , and given two scalars s ∈ (0 , 1 4 ) and f ∈ ( 3 4 , 1) , then the largest initialization value for which the learning phases of the values are ( s, f ) -incremental, denoted σ th 0 , is bounded in the following way:

Original: [Equation]
Revised: [Equation]

Original: Proof sketch (the full proof is given in appendix A).
Revised: Proof sketch (the full proof is given in appendix A).

Original: Rewriting the separable differential equation in equation 8 to calculate the time until σ ( t )
Revised: Rewriting the separable differential equation in (4) to calculate the time until σ ( t )

Original: = ασ ∗ , we get the following:
Revised: = ασ ∗ , we get the following:

Original: [Equation]
Revised: [Equation]

Original: The condition for incremental learning is then the requirement that t f ( σ i ) ≤ t s ( σ j ) , resulting in:
Revised: The condition for incremental learning is then the requirement that t f ( σ i ) ≤ t s ( σ j ) , resulting in:

Original: [Equation]
Revised: [Equation]

Original: We then relax/restrict the above condition to get a necessary/sufﬁcient condition on σ 0 , leading to a lower and upper bound on σ th 0 .
Revised: We then relax/restrict the above condition to get a necessary/sufﬁcient condition on σ 0 , leading to a lower and upper bound on σ th 0 .

Original: Note that the value determining the condition for incremental learning is σ ∗ i σ ∗ j - if two values are in the same order of magnitude, then their ratio will be close to 1 and we will need a small initialization to obtain incremental learning.
Revised: Note that the value determining the condition for incremental learning is σ ∗ i σ ∗ j - if two values are in the same order of magnitude, then their ratio will be close to 1 and we will need a small initialization to obtain incremental learning.

Original: The dependence on the ratio changes with depth, and is exponential for N = 2 .
Revised: The dependence on the ratio changes with depth, and is exponential for N = 2 .

Original: This means that incremental learning, while possible for shallow models, is difﬁcult to see in practice.
Revised: This means that incremental learning, while possible for shallow models, is difﬁcult to see in practice.

Original: This result explains why changing the initialization scale in ﬁgure 2 changes the dynamics of the N ≥ 3 models, while not changing the dynamics for N = 2 noticeably.
Revised: This result explains why changing the initialization scale in ﬁgure 2 changes the dynamics of the N ≥ 3 models, while not changing the dynamics for N = 2 noticeably.

Original: The next theorem extends part of our analysis to gradient descent, a more realistic setting than the inﬁnitesimal learning rate of gradient ﬂow:
Revised: The next theorem extends part of our analysis to gradient descent, a more realistic setting than the inﬁnitesimal learning rate of gradient ﬂow:

Original: Theorem 3.
Revised: Theorem 3.

Original: Given two values σ i , σ j of a depth-2 toy linear model as in equation 1, such that σ ∗ i σ ∗ j = r > 1 and the model is initialized as in equation 4, and given two scalars s ∈ (0 , 14 ) and f ∈ ( 3 4 , 1) , and assuming σ ∗ j ≥ 2 σ 0 , and assuming we optimize with gradient descent with a learning rate η ≤ cσ ∗ 1 for c < 2( √ 2 − 1) and σ ∗ 1 the largest value of σ ∗ , then the largest initialization value for which the learning phases of the values are ( s, f ) -incremental, denoted σ th 0 , is lower and upper bounded in the following way:
Revised: Given two values σ i , σ j of a depth-2 toy linear model as in (1) , such that σ ∗ i σ ∗ j = r >and the model is initialized as in (3) , and given two scalars s ∈ (0 , 14 ) and f ∈ ( 34 , 1) , and assuming σ ∗ j ≥ 2 σ 0 , and assuming we optimize with gradient descent with a learning rate η ≤ cσ ∗ 1 for c < 2( √ 2 − 1) and σ ∗ 1 the largest value of σ ∗ , then the largest initialization value for which the learning phases of the values are ( s, f ) -incremental, denoted σ th 0 , is lower and upper bounded in the following way:

Original: [Equation]
Revised: [Equation]

Original: Where A and B are deﬁned as:
Revised: Where A and B are deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: We defer the proof to appendix B.
Revised: We defer the proof to appendix B.

Original: Note that this result, while less elegant than the bounds of the gradient ﬂow analysis, is similar in nature.
Revised: Note that this result, while less elegant than the bounds of the gradient ﬂow analysis, is similar in nature.

Original: Both A and B simplify to r when we take their ﬁrst order approximation around c = 0 , giving us similar bounds and showing that the condition on σ 0 for N = 2 is exponential in gradient descent as well.
Revised: Both A and B simplify to r when we take their ﬁrst order approximation around c = 0 , giving us similar bounds and showing that the condition on σ 0 for N = 2 is exponential in gradient descent as well.

Original: While similar gradient descent results are harder to obtain for deeper models, we discuss the general effect of depth on the gradient decent dynamics in appendix C.
Revised: While similar gradient descent results are harder to obtain for deeper models, we discuss the general effect of depth on the gradient decent dynamics in appendix C.

Original:  I NCREMENTAL L EARNING IN L ARGER M ODELS
Revised:  I NCREMENTAL L EARNING IN L ARGER M ODELS

Original: So far, we have only shown interesting properties of incremental learning caused by depth for a toy model.
Revised: So far, we have only shown interesting properties of incremental learning caused by depth for a toy model.

Original: In this section, we will relate several deep models to our toy model and show how incremental learning presents itself in larger models as well.
Revised: In this section, we will relate several deep models to our toy model and show how incremental learning presents itself in larger models as well.

Original: 4. M ATRIX S ENSING
Revised: 4. M ATRIX S ENSING

Original: The task of matrix sensing is a generalization of matrix completion, where our input space is X = R d × d
Revised: The task of matrix sensing is a generalization of matrix completion, where our input space is X = R d × d

Original: and our model is a matrix W ∈ R d × d , such that:
Revised: and our model is a matrix W ∈ R d × d , such that:

Original: [Equation]
Revised: [Equation]

Original: Following Arora et al.
Revised: Following Arora et al.

Original: (2019), we introduce depth by parameterizing the model using a product of matrices and the following initialization scheme ( W i ∈ R
Revised: (2019), we introduce depth by parameterizing the model using a product of matrices and the following initialization scheme ( W i ∈ R

Original: d × d ):
Revised: d × d ):

Original: [Equation]
Revised: [Equation]

Original: Note that when d = 1 , the deep matrix sensing model reduces to our toy model without weight sharing.
Revised: Note that when d = 1 , the deep matrix sensing model reduces to our toy model without weight sharing.

Original: We study the dynamics of the model under gradient ﬂow over a depth-normalized squared loss, assuming the data is labeled by a matrix sensing model parameterized by a PSD W ∗ ∈ R d × d :
Revised: We study the dynamics of the model under gradient ﬂow over a depth-normalized squared loss, assuming the data is labeled by a matrix sensing model parameterized by a PSD W ∗ ∈ R d × d :

Original: [Equation]
Revised: [Equation]

Original: The following theorem relates the deep matrix sensing model to our toy model, showing the two have the same dynamical equations:
Revised: The following theorem relates the deep matrix sensing model to our toy model, showing the two have the same dynamical equations:

Original: Theorem 4.
Revised: Theorem 4.

Original: Optimizing the deep matrix sensing model described in equation 12 with gradient ﬂow over the depth normalized squared loss (equation 14), with weights initialized as in equationleads to the following dynamical equations for different values of N :
Revised: Optimizing the deep matrix sensing model described in (5) with gradient ﬂow over the depth normalized squared loss ( (6) ), with weights initialized as in (5) leads to the following dynamical equations for different values of N :

Original: [Equation]
Revised: [Equation]

Original: Where σ i and σ ∗ i are the i th singular values of W and W ∗ , respectively, corresponding to the same singular vector.
Revised: Where σ i and σ ∗ i are the i th singular values of W and W ∗ , respectively, corresponding to the same singular vector.

Original: The proof follows that of Saxe et al.
Revised: The proof follows that of Saxe et al.

Original: and Gidel et al.
Revised: and Gidel et al.

Original: and is deferred to appendix E.
Revised: and is deferred to appendix E.

Original: Theorem 4 shows us that the bias towards sparse solutions introduced by depth in the toy model is equivalent to the bias for low-rank solutions in the matrix sensing task.
Revised: Theorem 4 shows us that the bias towards sparse solutions introduced by depth in the toy model is equivalent to the bias for low-rank solutions in the matrix sensing task.

Original: This bias was studied in a more general setting in Arora et al.
Revised: This bias was studied in a more general setting in Arora et al.

Original: (2019), with empirical results supporting the effect of depth on the obtainment of low-rank solutions under a more natural loss and initialization scheme.
Revised: (2019), with empirical results supporting the effect of depth on the obtainment of low-rank solutions under a more natural loss and initialization scheme.

Original: We recreate and discuss these experiments and their connection to our analysis in appendix E, and an example of these dynamics in deep matrix sensing can also be seen in panel (a) of ﬁgure 1.
Revised: We recreate and discuss these experiments and their connection to our analysis in appendix E, and an example of these dynamics in deep matrix sensing can also be seen in panel (a) of ﬁgure 1.

Original: 4. Q UADRATIC N EURAL N ETWORKS
Revised: 4. Q UADRATIC N EURAL N ETWORKS

Original: By drawing connections between quadratic networks and matrix sensing (as in Soltanolkotabi et al.
Revised: By drawing connections between quadratic networks and matrix sensing (as in Soltanolkotabi et al.

Original: (2018)), we can extend our results to these nonlinear models.
Revised: (2018)), we can extend our results to these nonlinear models.

Original: We will study a simpliﬁed quadratic network, where our input space is X = R d and the ﬁrst layer is parameterized by a weight matrix
Revised: We will study a simpliﬁed quadratic network, where our input space is X = R d and the ﬁrst layer is parameterized by a weight matrix

Original: W ∈ R d × d and followed by a quadratic activation function.
Revised: W ∈ R d × d and followed by a quadratic activation function.

Original: The ﬁnal layer will be a summation layer.
Revised: The ﬁnal layer will be a summation layer.

Original: We assume, like before, that the labeling function is a quadratic network parameterized by
Revised: We assume, like before, that the labeling function is a quadratic network parameterized by

Original: W ∗ ∈ R d × d .
Revised: W ∗ ∈ R d × d .

Original: Our model can be written in the following way, using the following orthogonal initialization scheme:
Revised: Our model can be written in the following way, using the following orthogonal initialization scheme:

Original: [Equation]
Revised: [Equation]

Original: Immediately, we see the similarity of the quadratic network to the deep matrix sensing model with
Revised: Immediately, we see the similarity of the quadratic network to the deep matrix sensing model with

Original: N = 2 , where the input space is made up of rank-1 matrices.
Revised: N = 2 , where the input space is made up of rank-1 matrices.

Original: However, the change in input space forces us to optimize over a different loss function to reproduce the same dynamics:
Revised: However, the change in input space forces us to optimize over a different loss function to reproduce the same dynamics:

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Given an input distribution over an input space X with a labeling function y : X → R and a hypothesis h , the variance loss is deﬁned in the following way:
Revised: Given an input distribution over an input space X with a labeling function y : X → R and a hypothesis h , the variance loss is deﬁned in the following way:

Original: [Equation]
Revised: [Equation]

Original: Note that minimizing this loss function amounts to minimizing the variance of the error, while the squared loss minimizes the second moment of the error.
Revised: Note that minimizing this loss function amounts to minimizing the variance of the error, while the squared loss minimizes the second moment of the error.

Original: We note that both loss functions have the same minimum for our problem, and the dynamics of the squared loss can be approximated in certain cases by the dynamics of the variance loss.
Revised: We note that both loss functions have the same minimum for our problem, and the dynamics of the squared loss can be approximated in certain cases by the dynamics of the variance loss.

Original: For a complete discussion of the two losses, including the cases where the two losses have similar dynamics, we refer the reader to appendix F.
Revised: For a complete discussion of the two losses, including the cases where the two losses have similar dynamics, we refer the reader to appendix F.

Original: Theorem 5.
Revised: Theorem 5.

Original: Minimizing the quadratic network described and initialized as in equation 16 with gradient ﬂow over the variance loss deﬁned in equation 2 leads to the following dynamical equations:
Revised: Minimizing the quadratic network described and initialized as in (7) with gradient ﬂow over the variance loss deﬁned in (2) leads to the following dynamical equations:

Original: [Equation]
Revised: [Equation]

Original: Where σ i and σ ∗ i are the i th singular values of W and W ∗ , respectively, corresponding to the same singular vector.
Revised: Where σ i and σ ∗ i are the i th singular values of W and W ∗ , respectively, corresponding to the same singular vector.

Original: We defer the proof to appendix F and note that these dynamics are the same as our depth-2 toy model, showing that shallow quadratic networks can exhibit incremental learning (albeit requiring a small initialization).
Revised: We defer the proof to appendix F and note that these dynamics are the same as our depth-2 toy model, showing that shallow quadratic networks can exhibit incremental learning (albeit requiring a small initialization).

Original: 4. D IAGONAL /C ONVOLUTIONAL L INEAR N ETWORKS
Revised: 4. D IAGONAL /C ONVOLUTIONAL L INEAR N ETWORKS

Original: While incremental learning has been described for deep linear networks in the past, it has been restricted to regression tasks.
Revised: While incremental learning has been described for deep linear networks in the past, it has been restricted to regression tasks.

Original: Here, we illustrate how incremental learning presents itself in binary classiﬁcation, where implicit bias results have so far focused on convergence at t → ∞ (Soudry et al.
Revised: Here, we illustrate how incremental learning presents itself in binary classiﬁcation, where implicit bias results have so far focused on convergence at t → ∞ (Soudry et al.

Original: (2018), Nacson et al.
Revised: (2018), Nacson et al.

Original: , Ji & Telgarsky (2019)).
Revised: , Ji & Telgarsky (2019)).

Original: Deep linear networks with diagonal weight matrices have been shown to be biased towards sparse solutions when N > 1 in Gunasekar et al.
Revised: Deep linear networks with diagonal weight matrices have been shown to be biased towards sparse solutions when N > 1 in Gunasekar et al.

Original: (2018), and biased towards the max-margin solution for N = 1 .
Revised: (2018), and biased towards the max-margin solution for N = 1 .

Original: Instead of analyzing convergence at t → ∞ , we intend to show that the model favors sparse solutions for the entire duration of optimization, and that this is due to the dynamics of incremental learning.
Revised: Instead of analyzing convergence at t → ∞ , we intend to show that the model favors sparse solutions for the entire duration of optimization, and that this is due to the dynamics of incremental learning.

Original: Our theoretical illustration will use our toy model as in equation 1 (initialized as in equation 4) as a special weight-shared case of deep networks with diagonal weight matrices, and we will then show empirical results for the more general setting.
Revised: Our theoretical illustration will use our toy model as in (1) (initialized as in (3)) as a special weightshared case of deep networks with diagonal weight matrices, and we will then show empirical results for the more general setting.

Original: We analyze the optimization dynamics of this model over a separable dataset { x i , y i } mi =1 where y i ∈ {± 1 } .
Revised: We analyze the optimization dynamics of this model over a separable dataset { x i , y i } mi =1 where y i ∈ {± 1 } .

Original: We use the exponential loss ( (cid:96) ( f ( x ) , y ) =
Revised: We use the exponential loss ( (cid:96) ( f ( x ) , y ) =

Original: e − yf ( x ) ) for the theoretical illustration and experiment on the exponential and logistic losses.
Revised: e − yf ( x ) ) for the theoretical illustration and experiment on the exponential and logistic losses.

Original: Computing the gradient for the model over w , the gradient ﬂow dynamics for σ become:
Revised: Computing the gradient for the model over w , the gradient ﬂow dynamics for σ become:

Original: [Equation]
Revised: [Equation]

Original: We see the same dynamical attenuation of small values of σ that is seen in the regression model, caused by the multiplication by σ 2 − 2 N i .
Revised: We see the same dynamical attenuation of small values of σ that is seen in the regression model, caused by the multiplication by σ 2 − 2 N i .

Original: From this, we can expect the same type of incremental learning to occur - weights of σ will be learned incrementally until the dataset can be separated by the current support of σ .
Revised: From this, we can expect the same type of incremental learning to occur - weights of σ will be learned incrementally until the dataset can be separated by the current support of σ .

Original: Then, the dynamics strengthen the growth of the current support while relatively attenuating that of the other values.
Revised: Then, the dynamics strengthen the growth of the current support while relatively attenuating that of the other values.

Original: Since the data is separated, increasing the values of the current support reduces the loss and the magnitude of subsequent gradients, and so we should expect the support to remain the same and the model to converge to a sparse solution.
Revised: Since the data is separated, increasing the values of the current support reduces the loss and the magnitude of subsequent gradients, and so we should expect the support to remain the same and the model to converge to a sparse solution.

Original: Granted, the above description is just intuition, but panel (c) of ﬁgure 1 shows how it is born out in practice (similar results are obtained for the logistic loss).
Revised: Granted, the above description is just intuition, but panel (c) of ﬁgure 1 shows how it is born out in practice (similar results are obtained for the logistic loss).

Original: In appendix G we further explore this model, showing deeper networks have a stronger bias for sparsity.
Revised: In appendix G we further explore this model, showing deeper networks have a stronger bias for sparsity.

Original: We also observe that the initialization scale plays a similar role as before - deep models are less biased towards sparsity when σ 0 is large.
Revised: We also observe that the initialization scale plays a similar role as before - deep models are less biased towards sparsity when σ 0 is large.

Original: In their work, Gunasekar et al.
Revised: In their work, Gunasekar et al.

Original: (2018) show an equivalence between the diagonal network and the circular-convolutional network in the frequency domain.
Revised: (2018) show an equivalence between the diagonal network and the circular-convolutional network in the frequency domain.

Original: According to their results, we should expect to see the same sparsity-bias of diagonal networks in convolutional networks, when looking at the Fourier coefﬁcients of σ .
Revised: According to their results, we should expect to see the same sparsity-bias of diagonal networks in convolutional networks, when looking at the Fourier coefﬁcients of σ .

Original: An example of this can be seen in panel (d) of ﬁgure 1, and we refer the reader to appendix G for a full discussion of their convolutional model and it’s incremental learning dynamics.
Revised: An example of this can be seen in panel (d) of ﬁgure 1, and we refer the reader to appendix G for a full discussion of their convolutional model and it’s incremental learning dynamics.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Gradient-based optimization for deep linear models has an implicit bias towards simple (sparse) solutions, caused by an incremental search strategy over the hypothesis space.
Revised: Gradient-based optimization for deep linear models has an implicit bias towards simple (sparse) solutions, caused by an incremental search strategy over the hypothesis space.

Original: Deeper models have a stronger tendency for incremental learning, exhibiting it in more realistic initialization scales.
Revised: Deeper models have a stronger tendency for incremental learning, exhibiting it in more realistic initialization scales.

Original: This dynamical phenomenon exists for the entire optimization process for regression as well as classiﬁcation tasks, and for many types of models - diagonal networks, convolutional networks, matrix completion and even the nonlinear quadratic network.
Revised: This dynamical phenomenon exists for the entire optimization process for regression as well as classiﬁcation tasks, and for many types of models - diagonal networks, convolutional networks, matrix completion and even the nonlinear quadratic network.

Original: We believe this kind of dynamical analysis may be able to shed light on the generalization of deeper nonlinear neural networks as well, with shallow quadratic networks being only a ﬁrst step towards that goal. 
Revised: We believe this kind of dynamical analysis may be able to shed light on the generalization of deeper nonlinear neural networks as well, with shallow quadratic networks being only a ﬁrst step towards that goal. A CKNOWLEDGMENTS

Revised: This research is supported by the European Research Council (TheoryDL project).

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: Theorem.
Revised: Theorem.

Original: Given two values σ i , σ j of a toy linear model as in equation 1, such that σ ∗ i σ ∗ j = r >and the model is initialized as in equation 4, and given two scalars s ∈ (0 , 14 ) and f ∈ ( 34 , 1) , then the largest initialization value for which the learning phases of the values are ( s, f ) -incremental, denoted σ th 0 , is lower and upper bounded in the following way:
Revised: Given two values σ i , σ j of a toy linear model as in (1) , such that σ ∗ i σ ∗ j = r > 1 and the model is initialized as in (3) , and given two scalars s ∈ (0 , 14 ) and f ∈ ( 34 , 1) , then the largest initialization value for which the learning phases of the values are ( s, f ) -incremental, denoted σ th 0 , is lower and upper bounded in the following way:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Our strategy will be to deﬁne the time t α for which a value reaches a fraction α of it’s optimal value, and then require that t f ( σ i ) ≤ t s ( σ j ) .
Revised: Our strategy will be to deﬁne the time t α for which a value reaches a fraction α of it’s optimal value, and then require that t f ( σ i ) ≤ t s ( σ j ) .

Original: We begin with recalling the differential equation which determines the dynamics of the model:
Revised: We begin with recalling the differential equation which determines the dynamics of the model:

Original: [Equation]
Revised: [Equation]

Original: Since the solution for N ≥ 3 is implicit and difﬁcult to manage in a general form, we will deﬁne t α using the integral of the differential equation.
Revised: Since the solution for N ≥ 3 is implicit and difﬁcult to manage in a general form, we will deﬁne t α using the integral of the differential equation.

Original: The equation is separable, and under initialization of σ 0 we can describe t α ( σ ) in the following way:[Equation]
Revised: The equation is separable, and under initialization of σ 0 we can describe t α ( σ ) in the following way:

Original: [Equation]
Revised: [Equation]

Original: Incremental learning takes place when σ i ( t f )
Revised: Incremental learning takes place when σ i ( t f )

Original: = fσ ∗ i happens before σ j ( t s ) =
Revised: = fσ ∗ i happens before σ j ( t s ) =

Original: sσ ∗ j .
Revised: sσ ∗ j .

Original: We can write this condition in the following way:
Revised: We can write this condition in the following way:

Original: [Equation]
Revised: [Equation]

Original: Plugging in σ i = rσ j and rearranging, we get the following necessary and sufﬁcient condition for incremental learning:
Revised: Plugging in σ i = rσ j and rearranging, we get the following necessary and sufﬁcient condition for incremental learning:

Original: [Equation]
Revised: [Equation]

Original: Our last step before relaxing and restricting our condition will be to split the integral on the left-hand side into two integrals:
Revised: Our last step before relaxing and restricting our condition will be to split the integral on the left-hand side into two integrals:

Original: [Equation]
Revised: [Equation]

Original: At this point, we cannot solve this equation and isolate σ 0 to obtain a clear threshold condition on it for incremental learning.
Revised: At this point, we cannot solve this equation and isolate σ 0 to obtain a clear threshold condition on it for incremental learning.

Original: Instead, we will relax/restrict the above condition to get a necessary/sufﬁcient condition on σ 0 , leading to a lower and upper bound on the threshold value of σ 0 .
Revised: Instead, we will relax/restrict the above condition to get a necessary/sufﬁcient condition on σ 0 , leading to a lower and upper bound on the threshold value of σ 0 .

Original: S UFFICIENT C ONDITION
Revised: S UFFICIENT C ONDITION

Original: To obtain a sufﬁcient (but not necessary) condition on σ 0 , we may make the condition stricter either by increasing the left-hand side or decreasing the right-hand side.
Revised: To obtain a sufﬁcient (but not necessary) condition on σ 0 , we may make the condition stricter either by increasing the left-hand side or decreasing the right-hand side.

Original: We can increase the left-hand side by removing  from left ) and then combine the left most right
Revised: We can increase the left-hand side by removing r from the left-most integral’s denominator ( r > 1 ) and then combine the left-most and right-most integrals:

Original: [Equation]
Revised: [Equation]

Original: Next, we note that the integration bounds give us a bound on σ for either integral.
Revised: Next, we note that the integration bounds give us a bound on σ for either integral.

Original: This means we can replace 1 − σσ ∗ j with 1 on the right-hand side, and replace 1 − σrσ ∗ j with 1 − f on the left-hand side:
Revised: This means we can replace 1 − σσ ∗ j with 1 on the right-hand side, and replace 1 − σrσ ∗ j with 1 − f on the left-hand side:

Original: [Equation]
Revised: [Equation]

Original: We may now solve these integrals for every N and isolate σ 0 , obtaining the lower bound on σ th 0 .
Revised: We may now solve these integrals for every N and isolate σ 0 , obtaining the lower bound on σ th 0 .

Original: We start with the case where N = 2 :
Revised: We start with the case where N = 2 :

Original: [Equation]
Revised: [Equation]

Original: Rearranging to isolate σ 0 , we obtain our result:
Revised: Rearranging to isolate σ 0 , we obtain our result:

Original: [Equation]
Revised: [Equation]

Revised: For the N ≥ 3 case

Original: [Equation]
Revised: [Equation]

Original: For the N ≥ 3 case, we have the following after solving the integrals:
Revised: we have the following after solving the integrals:

Original: [Equation]
Revised: [Equation]

Original: For simplicity we may further restrict the condition by removing the term (cid:0) 1 rfσ ∗ j (cid:1) 1 − 2 N .
Revised: For simplicity we may further restrict the condition by removing the term (cid:0) 1 rfσ ∗ j (cid:1) 1 − 2 N .

Original: Solving for σ 0 gives us the following:
Revised: Solving for σ 0 gives us the following:

Original: [Equation]
Revised: [Equation]

Original: N ECESSARY C ONDITION
Revised: N ECESSARY C ONDITION

Original: To obtain a necessary (but not sufﬁcient) condition on σ 0 , we may relax the condition in equationeither by decreasing the left-hand side or increasing the right-hand side.
Revised: To obtain a necessary (but not sufﬁcient) condition on σ 0 , we may relax the condition in (8) either by decreasing the left-hand side or increasing the right-hand side.

Original: We begin by rearranging the equation:
Revised: We begin by rearranging the equation:

Original: [Equation]
Revised: [Equation]

Original: Like before, we may use the integration bounds to bound σ .
Revised: Like before, we may use the integration bounds to bound σ .

Original: Plugging in σ =
Revised: Plugging in σ =

Original: sσ ∗ j for all integrals decreases the left-hand side and increases the right-hand side, leading us to the following:
Revised: sσ ∗ j for all integrals decreases the left-hand side and increases the right-hand side, leading us to the following:

Original: [Equation]
Revised: [Equation]

Original: Rearranging, we get the following inequality:
Revised: Rearranging, we get the following inequality:

Original: [Equation]
Revised: [Equation]

Original: We now solve the integrals for the different cases.
Revised: We now solve the integrals for the different cases.

Original: For N = 2 , we have:
Revised: For N = 2 , we have:

Original: [Equation]
Revised: [Equation]

Original: Rearranging to isolate σ 0 , we get our condition:
Revised: Rearranging to isolate σ 0 , we get our condition:

Original: [Equation]
Revised: [Equation]

Original: Finally, for N ≥ 3 , we solve the integrals to give us:
Revised: Finally, for N ≥ 3 , we solve the integrals to give us:

Original: [Equation]
Revised: [Equation]

Original: Rearranging to isolate σ 0 , we get our condition:
Revised: Rearranging to isolate σ 0 , we get our condition:

Original: [Equation]
Revised: [Equation]

Original: S UMMARY
Revised: S UMMARY

Original: For a given N , we derived a sufﬁcient condition and a necessary condition on σ 0 for ( s, f )- incremental learning.
Revised: For a given N , we derived a sufﬁcient condition and a necessary condition on σ 0 for ( s, f )- incremental learning.

Original: The necessary and sufﬁcient condition on σ 0 , which is the largest initialization value for which we see incremental learning (denoted σ th 0 ), is between the two derived bounds.
Revised: The necessary and sufﬁcient condition on σ 0 , which is the largest initialization value for which we see incremental learning (denoted σ th 0 ), is between the two derived bounds.

Original: The precise bounds can possibly be improved a bit, but the asymptotic dependence on r is the crux of the matter, showing the dependence on r changes with depth with a substantial difference when we move from shallow models ( N = 2 ) to deeper ones ( N ≥ 3 )
Revised: The precise bounds can possibly be improved a bit, but the asymptotic dependence on r is the crux of the matter, showing the dependence on r changes with depth with a substantial difference when we move from shallow models ( N = 2 ) to deeper ones ( N ≥ 3 )

Original: B P ROOF OF T HEOREM
Revised: B P ROOF OF T HEOREM

Original: Theorem.
Revised: Theorem.

Original: Given two values σ i , σ j of a depth-2 toy linear model as in equation 1, such that σ ∗ i σ ∗ j = r > 1 and the model is initialized as in equation 4, and given two scalars s ∈ (0 , 1 4 ) and f ∈ ( 3 4 , 1) , and assuming σ ∗ j ≥ 2 σ 0 , and assuming we optimize with gradient descent with a learning rate η ≤ cσ ∗ 1 ≤ 2( [Equation] 1)  σ ∗, then the largest initialization value for which the learning phases of the values are ( s, f ) -incremental, denoted σ th 0 , is lower and upper bounded in the following way:
Revised: Given two values σ i , σ j of a depth-2 toy linear model as in (1) , such that σ ∗ i σ ∗ j = r >and the model is initialized as in (3) , and given two scalars s ∈ (0 , 14 ) and f ∈ ( 34 , 1) , and assuming σ ∗ j ≥ 2 σ 0 , and assuming we optimize with gradient descent with a learning rate η ≤ cσ ∗ 1 for c < 2( √ 2 − 1) and σ ∗ 1 the largest value of σ ∗ , then the largest initialization value for which the learning phases of the values are ( s, f ) -incremental, denoted σ th 0 , is lower and upper bounded in the following way:

Original: [Equation]
Revised: [Equation]

Original: Where A and B are deﬁned as:
Revised: Where A and B are deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: To show our result for gradient descent and N = 2 , we build on the proof techniques of theorem 3 of Gidel et al.
Revised: To show our result for gradient descent and N = 2 , we build on the proof techniques of theorem 3 of Gidel et al.

Original: We start by deriving the recurrence relation for the values σ ( t ) for general depth, when t now stands for the iteration.
Revised: We start by deriving the recurrence relation for the values σ ( t ) for general depth, when t now stands for the iteration.

Original: Remembering that w ni = σ i , we write down the gradient update for w i ( t ) :
Revised: Remembering that w ni = σ i , we write down the gradient update for w i ( t ) :

Original: [Equation]
Revised: [Equation]

Original: Raising w i ( t ) to the N th power, we get the gradient update for the σ values:
Revised: Raising w i ( t ) to the N th power, we get the gradient update for the σ values:

Original: [Equation]
Revised: [Equation]

Original: Next, we will prove a simple lemma which gives us the maximal learning rate we will consider for the analysis, for which there is no overshooting (the values don’t grow larger than the optimal values).
Revised: Next, we will prove a simple lemma which gives us the maximal learning rate we will consider for the analysis, for which there is no overshooting (the values don’t grow larger than the optimal values).

Original: Lemma 1.
Revised: Lemma 1.

Original: For the gradient update in equation 27, assuming σ i (0) <
Revised: For the gradient update in (9) , assuming σ i (0) <

Original: σ ∗ i , if η ≤ (cid:16) 1 σ ∗ 1 (cid:17) 2 − 2 N and N ≥ 2 , then:
Revised: σ ∗ i , if η ≤ (cid:16) 1 σ [Equation] 1 (cid:17) 2 − 2 N and N ≥ 2 , then:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Plugging in η = c (cid:16) 1 σ ∗ i (cid:17)
Revised: Plugging in η = c (cid:16) 1 σ ∗ i (cid:17)

Original: 2 − 2 N for c ≤ 1 , we have:
Revised: 2 − 2 N for c ≤ 1 , we have:

Original: [Equation]
Revised: [Equation]

Original: Deﬁning r i =
Revised: Deﬁning r i =

Original: σ i σ ∗ i and dividing both sides by σ ∗ i , we have:
Revised: σ i σ ∗ i and dividing both sides by σ ∗ i , we have:

Original: [Equation]
Revised: [Equation]

Original: It is enough to show that for any 0 ≤ r ≤ 1 , we have that re r 1 − 2 N (1 − r ) ≤ 1 , as over-shooting occurs when r i ( t ) > 1 .
Revised: It is enough to show that for any 0 ≤ r ≤ 1 , we have that re r 1 − 2 N (1 − r ) ≤ 1 , as over-shooting occurs when r i ( t ) > 1 .

Original: Indeed, this function is monotonic increasing in 0 ≤ r ≤ 1 (since the exponent is non negative), and equals 1 when r = 1 .
Revised: Indeed, this function is monotonic increasing in 0 ≤ r ≤ 1 (since the exponent is non negative), and equals 1 when r = 1 .

Original: Since r = 1 is a ﬁxed point and no iteration that starts at r < 1 can cross 1 , then r i ( t ) ≤ 1 for any t .
Revised: Since r = 1 is a ﬁxed point and no iteration that starts at r < 1 can cross 1 , then r i ( t ) ≤ 1 for any t .

Original: This concludes our proof.
Revised: This concludes our proof.

Original: Under this choice of learning rate, we can now obtain our incremental learning results for gradient descent when N = 2 .
Revised: Under this choice of learning rate, we can now obtain our incremental learning results for gradient descent when N = 2 .

Original: Our strategy will be bounding σ i ( t ) from below and above, which will give us a lower and upper bound for t α ( σ i ) .
Revised: Our strategy will be bounding σ i ( t ) from below and above, which will give us a lower and upper bound for t α ( σ i ) .

Original: Once we have these bounds, we will be able to describe either a necessary or a sufﬁcient condition on σ 0 for incremental learning, similar to theorem 2.
Revised: Once we have these bounds, we will be able to describe either a necessary or a sufﬁcient condition on σ 0 for incremental learning, similar to theorem 2.

Original: The update rule for N = 2 is:
Revised: The update rule for N = 2 is:

Original: [Equation]
Revised: [Equation]

Original: Next, we plug in η = c σ ∗ 1 for c < 2( √ 2 − 1) < 1 and denote R i =
Revised: Next, we plug in η = cσ ∗ 1 for c < 2( √ 2 − 1) < 1 and denote R i =

Original: σ ∗ i σ ∗ 1 ≤ 1 and r i ( t ) =
Revised: σ ∗ i σ ∗ 1 ≤ 1 and r i ( t ) =

Original: σ i ( t ) σ ∗ i ≤ 1 to get:
Revised: σ i ( t ) σ ∗ i ≤ 1 to get:

Original: [Equation]
Revised: [Equation]

Original: Following theorem 3 of Gidel et al.
Revised: Following theorem 3 of Gidel et al.

Original: (2019), we bound
Revised: (2019), we bound

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Where in the fourth line we use the inequality 11+ x ≥ 1 − x, ∀ x ≥ 0 .
Revised: Where in the fourth line we use the inequality 11+ x ≥ 1 − x, ∀ x ≥ 0 .

Original: We may now subtract 1 σ ∗ i from both sides to obtain:
Revised: We may now subtract 1 σ ∗ i from both sides to obtain:

Original: [Equation]
Revised: [Equation]

Original: We may now obtain a bound on t α ( σ i ) by plugging in σ i ( t ) =
Revised: We may now obtain a bound on t α ( σ i ) by plugging in σ i ( t ) =

Original: ασ ∗ i and taking the log:
Revised: ασ ∗ i and taking the log:

Original: [Equation]
Revised: [Equation]

Original: Rearranging (note that log (cid:0) 1 − cR
Revised: Rearranging (note that log (cid:0) 1 − cR

Original: i − c 2 4 R 2 i (cid:1) < 0 and that our choice of c keeps the argument of the log positive), we get:
Revised: i − c 2 4 R 2 i (cid:1) < 0 and that our choice of c keeps the argument of the log positive), we get:

Original: [Equation]
Revised: [Equation]

Original: Next [Equation] we follow the same procedure for an upper bound.
Revised: Next, we follow the same procedure for an upper bound.

Original: Starting with our update step:
Revised: Starting with our update step:

Original: [Equation]
Revised: [Equation]

Original: Where in the last line we use the inequality 11+ x ≤ 1 − x + x 2 , ∀ x ≥ 0 .
Revised: Where in the last line we use the inequality 11+ x ≤ 1 − x + x 2 , ∀ x ≥ 0 .

Original: Subtracting 1 σ ∗ i from both sides [Equation] we get:
Revised: Subtracting 1 σ ∗ i from both sides, we get:

Original: [Equation]
Revised: [Equation]

Original: Rearranging like before, we get the bound on the α -time:
Revised: Rearranging like before, we get the bound on the α -time:

Original: [Equation]
Revised: [Equation]

Original: Given these bounds, we would like to ﬁnd the conditions on σ 0 that allows for ( s, f ) -incremental learning.
Revised: Given these bounds, we would like to ﬁnd the conditions on σ 0 that allows for ( s, f ) -incremental learning.

Original: We will ﬁnd a sufﬁcient condition and a necessary condition, like in the proof of theorem 
Revised: We will ﬁnd a sufﬁcient condition and a necessary condition, like in the proof of theorem 2.

Original: 2. S UFFICIENT C ONDITION
Revised: S UFFICIENT C ONDITION

Original: A sufﬁcient condition for incremental learning will be one which is possibly stricter than the exact condition.
Revised: A sufﬁcient condition for incremental learning will be one which is possibly stricter than the exact condition.

Original: We can ﬁnd such a condition by requiring the upper bound of t f ( σ i ) to be smaller than the lower bound on t s ( σ j ) .
Revised: We can ﬁnd such a condition by requiring the upper bound of t f ( σ i ) to be smaller than the lower bound on t s ( σ j ) .

Original: This becomes the following condition:
Revised: This becomes the following condition:

Original: [Equation]
Revised: [Equation]

Original: Deﬁning A = log (cid:0) 
Revised: Deﬁning A = log (cid:0) 1 − cR i + c 2 R [Equation] i (cid:1)

Original: [Equation]
Revised: [Equation]

Original: and rearranging, we get the following:
Revised: and rearranging, we get the following:

Original: [Equation]
Revised: [Equation]

Original: We may now take the exponent of both sides and rearrange again, remembering σ ∗ i σ ∗ j = r > 1 , to get the following condition:
Revised: We may now take the exponent of both sides and rearrange again, remembering σ ∗ i σ ∗ j = r > 1 , to get the following condition:

Original: [Equation]
Revised: [Equation]

Original: Now, we will add the very reasonable assumption that σ ∗ j ≥ 2 σ 0 , which allows us to replace σ 
Revised: Now, we will add the very reasonable assumption that σ ∗ j ≥ 2 σ 0 , which allows us to replace σ ∗ j − [Equation] rσ

Original: [Equation]
Revised: [Equation]

Original: j − 
Revised: − σ

Original: [Equation]
Revised: [Equation]

Original: with 12 r and replace ( σ ∗ j − σ 0 )
Revised: with 12 r and replace ( σ ∗ j − σ 0 )

Original: A − 1 with ( 12 σ j )
Revised: A − 1 with ( 12 σ j )

Original: A − 1 , only making the condition stricter.
Revised: A − 1 , only making the condition stricter.

Original: This simpliﬁes the expression to the following:
Revised: This simpliﬁes the expression to the following:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now we can rearrange and isolate σ 0 to get a sufﬁcient condition for incremental learning:
Revised: Now we can rearrange and isolate σ 0 to get a sufﬁcient condition for incremental learning:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: N ECESSARY C ONDITION
Revised: N ECESSARY C ONDITION

Original: A necessary condition for incremental learning will be one which is possibly more relaxed than the exact condition.
Revised: A necessary condition for incremental learning will be one which is possibly more relaxed than the exact condition.

Original: We can ﬁnd such a condition by requiring the lower bound of t f ( σ i ) to be smaller than the upper bound on t s ( σ j ) .
Revised: We can ﬁnd such a condition by requiring the lower bound of t f ( σ i ) to be smaller than the upper bound on t s ( σ j ) .

Original: This becomes the following condition:
Revised: This becomes the following condition:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Deﬁning B = log (cid:0) 1 − cR
Revised: Deﬁning B = log (cid:0) 1 − cR

Original: i − c 2 4 R 2 i (cid:1) log (cid:0) 1 − cR j
Revised: i − c 2 4 R 2 i (cid:1) log (cid:0) 1 − cR j

Original: + c 2 R 2 j (cid:1) and rearranging, we get the following:
Revised: + c 2 R 2 j (cid:1) and rearranging, we get the following:

Original: [Equation]
Revised: [Equation]

Original: We may now take the exponent of both sides and rearrange again, remembering σ ∗ i σ ∗ j = r > 1 , to get the following condition:
Revised: We may now take the exponent of both sides and rearrange again, remembering σ ∗ i σ ∗ j = r > 1 , to get the following condition:

Original: [Equation]
Revised: [Equation]

Original: We may now relax the condition further, by removing the r from the denominator of the left-hand side and the σ 0 from the numerator.
Revised: We may now relax the condition further, by removing the r from the denominator of the left-hand side and the σ 0 from the numerator.

Original: This gives us the following:
Revised: This gives us the following:

Original: [Equation]
Revised: [Equation]

Original: Finally, rearranging gives us the necessary condition:
Revised: Finally, rearranging gives us the necessary condition:

Original: [Equation]
Revised: [Equation]

Original: C D ISCUSSION OF G RADIENT D ESCENT FOR G ENERAL N
Revised: C D ISCUSSION OF G RADIENT D ESCENT FOR G ENERAL N

Original: While we were able to generalize our result to gradient descent for N = 2 , our proof technique relies on the ability to get a non-implicit solution for σ ( t ) which we discretized and bounded.
Revised: While we were able to generalize our result to gradient descent for N = 2 , our proof technique relies on the ability to get a non-implicit solution for σ ( t ) which we discretized and bounded.

Original: This is harder to generalize to larger values of N , where the solution is implicit.
Revised: This is harder to generalize to larger values of N , where the solution is implicit.

Original: Still, we can informally illustrate the effect of depth on the dynamics of gradient descent by approximating the update rule of the values.
Revised: Still, we can informally illustrate the effect of depth on the dynamics of gradient descent by approximating the update rule of the values.

Original: We start by reminding ourselves of the gradient descent update rule for σ , for a learning rate η = c 
Revised: We start by reminding ourselves of the gradient descent update rule for σ , for a learning rate η = c (cid:0)

Original: [Equation]
Revised: [Equation]

Original: To compare two values in the same scales, we will divide both sides by the optimal value σ ∗ i and look at the update step of the ratio r i =
Revised: To compare two values in the same scales, we will divide both sides by the optimal value σ ∗ i and look at the update step of the ratio r i =

Original: σ i ( t ) σ ∗ i , also denoting R i = σ∗ i σ
Revised: σ i ( t ) σ ∗ i , also denoting R i = σ

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We will focus on the early stages of the optimization process, where r (cid:28) 1 .
Revised: We will focus on the early stages of the optimization process, where r (cid:28) 1 .

Original: This means we can neglect the 1 − r i ( t ) term in the update step, giving us the approximate update step we will use to compare the general i, j values:
Revised: This means we can neglect the 1 − r i ( t ) term in the update step, giving us the approximate update step we will use to compare the general i, j values:

Original: [Equation]
Revised: [Equation]

Original: We would like to compare the dynamics of r i and r j , which is difﬁcult to do when the recurrence relation isn’t solvable.
Revised: We would like to compare the dynamics of r i and r j , which is difﬁcult to do when the recurrence relation isn’t solvable.

Original: However, we can observe the ﬁrst iteration of gradient descent and see how depth affects this iteration.
Revised: However, we can observe the ﬁrst iteration of gradient descent and see how depth affects this iteration.

Original: Since we are dealing with variables which are ratios of different optimal values, the initial values of r are different.
Revised: Since we are dealing with variables which are ratios of different optimal values, the initial values of r are different.

Original: Denoting r =
Revised: Denoting r =

Original: σ ∗ i σ ∗ j , we can describe the initialization of r j using that of r i :
Revised: σ ∗ i σ ∗ j , we can describe the initialization of r j using that of r i :

Original: [Equation]
Revised: [Equation]

Original: Plugging in the initial conditions and noting that R i =
Revised: Plugging in the initial conditions and noting that R i =

Original: r R j , we get:
Revised: r R j , we get:

Original: [Equation]
Revised: [Equation]

Original: We see that the two ratios have a similar update, with the ratio of optimal values playing a role in how large the initial value is versus how large the added value is.
Revised: We see that the two ratios have a similar update, with the ratio of optimal values playing a role in how large the initial value is versus how large the added value is.

Original: When we use a small learning rate, we have a very small c which means we can make a ﬁnal approximation and neglect the higher order terms of c :
Revised: When we use a small learning rate, we have a very small c which means we can make a ﬁnal approximation and neglect the higher order terms of c :

Original: [Equation]
Revised: [Equation]

Original: We can see that while the initial conditions favor r j , the size of the update for r i is larger by a factor of r N − 1 N when the initialization and learning rates are small.
Revised: We can see that while the initial conditions favor r j , the size of the update for r i is larger by a factor of r N − 1 N when the initialization and learning rates are small.

Original: This accumulates throughout the optimization, making r i eventually converge faster than r j .
Revised: This accumulates throughout the optimization, making r i eventually converge faster than r j .

Original: The effect of depth here is clear - the deeper the model, the larger the relative step size of r i and the faster it converges relative to r j .
Revised: The effect of depth here is clear - the deeper the model, the larger the relative step size of r i and the faster it converges relative to r j .

Original: D C OMPARISON OF T HE T OY M ODEL AND OMP
Revised: D C OMPARISON OF T HE T OY M ODEL AND OMP

Original: Learning our toy model, when it’s incremental learning is taken to the limit, can be described as an iterative procedure where at every step an additional feature is introduced such that it’s weight is non-zero and then the model is optimized over the current set of features.
Revised: Learning our toy model, when it’s incremental learning is taken to the limit, can be described as an iterative procedure where at every step an additional feature is introduced such that it’s weight is non-zero and then the model is optimized over the current set of features.

Original: This description is also relevant for the sparse approximation algorithm orthogonal matching pursuit (Pati et al., 1993), where the next feature is greedily chosen to be the one which most improves the current model.
Revised: This description is also relevant for the sparse approximation algorithm orthogonal matching pursuit (Pati et al., 1993), where the next feature is greedily chosen to be the one which most improves the current model.

Original: While the toy model and OMP are very different algorithms for learning sparse linear models, we will show empirically that they behave similarly.
Revised: While the toy model and OMP are very different algorithms for learning sparse linear models, we will show empirically that they behave similarly.

Original: This allows us to view incremental learning as a continuous-time extension of a greedy iterative algorithm.
Revised: This allows us to view incremental learning as a continuous-time extension of a greedy iterative algorithm.

Original: To allow for negative weights in our experiments, we augment our toy model as in the toy model of
Revised: To allow for negative weights in our experiments, we augment our toy model as in the toy model of

Original: Woodworth et al.
Revised: Woodworth et al.

Original: Our model will have the same induced form as before:
Revised: Our model will have the same induced form as before:

Original: [Equation]
Revised: [Equation]

Original: However, we parameterize σ using w + , w − ∈ R d in the following way:
Revised: However, we parameterize σ using w + , w − ∈ R d in the following way:

Original: [Equation]
Revised: [Equation]

Original: We can now treat this algorithm as a sparse approximation pursuit algorithm - given a dictionary
Revised: We can now treat this algorithm as a sparse approximation pursuit algorithm - given a dictionary

Original: D ∈ R d × n and an example x ∈ R d , we wish to ﬁnd the sparsest α for which
Revised: D ∈ R d × n and an example x ∈ R d , we wish to ﬁnd the sparsest α for which

Original: Dα
Revised: Dα

Original: ≈ x by minimizing the (cid:96) 0 norm of α subject to ||
Revised: ≈ x by minimizing the (cid:96) 0 norm of α subject to ||

Original: Dα − x || 22 = 0 3 .
Revised: Dα − x || 22 = 0 3 .

Original: Under this setting, we can compare OMP to our toy model by comparing the sets of features that the two algorithms choose for a given example and dictionary.
Revised: Under this setting, we can compare OMP to our toy model by comparing the sets of features that the two algorithms choose for a given example and dictionary.

Original: In ﬁgure 3 we run such a comparison.
Revised: In ﬁgure 3 we run such a comparison.

Original: Using a dictionary of 1000 atoms and an example of dimensionality 80 sampled from a random hidden vector of a given sparsity s , we run both algorithms and record the ﬁrst s features chosen 4 .
Revised: Using a dictionary of 1000 atoms and an example of dimensionality 80 sampled from a random hidden vector of a given sparsity s , we run both algorithms and record the ﬁrst s features chosen 4 .

Revised: For

Original: For every sparsity s , we plot the mean fraction of agreement between the sets of features chosen by OMP and the toy model over 100 experiments.
Revised: For every sparsity s , we plot the mean fraction of agreement between the sets of features chosen by OMP and the toy model over 100 experiments.

Original: We see that the two algorithms choose very similar features at the beginning, suggesting that the deep model approximates the discrete behavior of OMP.
Revised: We see that the two algorithms choose very similar features at the beginning, suggesting that the deep model approximates the discrete behavior of OMP.

Original: Only when the number of features increases do we see that the behavior of the two models begins to differ, caused by the fact that the toy model has a ﬁnite initialization scale and learning rate.
Revised: Only when the number of features increases do we see that the behavior of the two models begins to differ, caused by the fact that the toy model has a ﬁnite initialization scale and learning rate.

Original: These experiments demonstrate the similarity between the incremental learning of deep models and the discrete behavior of greedy approximation algorithms such as OMP.
Revised: These experiments demonstrate the similarity between the incremental learning of deep models and the discrete behavior of greedy approximation algorithms such as OMP.

Original: Adopting this view also allows us to deﬁne another strength of the dynamics of deep models - while greedy algorithms such as OMP require the analytical solution or approximation of every iterate, the dynamics of deep models are able to incrementally learn any differentiable function.
Revised: Adopting this view also allows us to put our ﬁnger on another strength of the dynamics of deep models - while greedy algorithms such as OMP require the analytical solution or approximation of every iterate, the dynamics of deep models are able to incrementally learn any differentiable function.

Original: For example, looking back at the matrix sensing task and the classiﬁcation models in section 4, we see that while there isn’t an immediate and efﬁcient extension of OMP for these settings, the dynamics of learning deep models extends naturally and exhibits the same incremental learning as OMP.
Revised: For example, looking back at the matrix sensing task and the classiﬁcation models in section 4, we see that while there isn’t an immediate and efﬁcient extension of OMP for these settings, the dynamics of learning deep models extends naturally and exhibits the same incremental learning as OMP.

Original: E I NCREMENTAL L EARNING IN M ATRIX S ENSING E. P ROOF OF T HEOREM
Revised: E I NCREMENTAL L EARNING IN M ATRIX S ENSING E. P ROOF OF T HEOREM

Original: Theorem.
Revised: Theorem.

Original: Minimizing the deep matrix sensing model described in equation 12 with gradient ﬂow over the depth normalized squared loss equation 14, with Gaussian inputs and weights initialized as in equation 12 leads to the following dynamical equations for different values of N :
Revised: Minimizing the deep matrix sensing model described in (5) with gradient ﬂow over the depth normalized squared loss (6) , with Gaussian inputs and weights initialized as in (5) leads to the following dynamical equations for different values of N :

Original: [Equation]
Revised: [Equation]

Original: Where σ i and σ ∗ i are the i th singular values of W and W ∗ , respectively, corresponding to the same singular vectors.
Revised: Where σ i and σ ∗ i are the i th singular values of W and W ∗ , respectively, corresponding to the same singular vectors.

Original: Proof.
Revised: Proof.

Original: We will adapt the proof from Saxe et al.
Revised: We will adapt the proof from Saxe et al.

Original: for multilayer linear networks.
Revised: for multilayer linear networks.

Original: The gradient ﬂow equations for W n , n ∈ [ N ] are:
Revised: The gradient ﬂow equations for W n , n ∈ [ N ] are:

Original: [Equation]
Revised: [Equation]

Original: Where we denote W j : k = (cid:81) ki
Revised: Where we denote W j : k = (cid:81) ki

Original: =
Revised: =

Original: j W i .
Revised: j W i .

Original: Since we assumed W ∗ is (symmetric) PSD, there exists an orthogonal matrix U for which D ∗ =
Revised: Since we assumed W ∗ is (symmetric) PSD, there exists an orthogonal matrix U for which D ∗ =

Original: UW ∗ U T where D ∗ is diagonal.
Revised: UW ∗ U T where D ∗ is diagonal.

Original: Under the initialization in equation 4, U diagonalizes all W n matrices at initialization such that D n = UW n U T = N √ σ 

Revised: Under the initialization in (3), U diagonalizes all W n matrices at initialization such that D n = UW n U T = N √ σ

Original: I .
Revised: I .

Original: Making this change of variables for all W n , we get:
Revised: Making this change of variables for all W n , we get:

Original: [Equation]
Revised: [Equation]

Original: Rearranging, we get a set of decoupled differential equations for the singular values of W n :
Revised: Rearranging, we get a set of decoupled differential equations for the singular values of W n :

Original: [Equation]
Revised: [Equation]

Original: Note that since these matrices are all diagonal at initialization, the above dynamics ensure that they remain diagonal throughout the optimization.
Revised: Note that since these matrices are all diagonal at initialization, the above dynamics ensure that they remain diagonal throughout the optimization.

Original: Denoting σ n,i as the i ’th singular value of W n and σ i as the i ’th singular value of W , we get the following differential equation:
Revised: Denoting σ n,i as the i ’th singular value of W n and σ i as the i ’th singular value of W , we get the following differential equation:

Original: [Equation]
Revised: [Equation]

Original: Since we assume at initialization that ∀ n, m, i : σ n,i (0)
Revised: Since we assume at initialization that ∀ n, m, i : σ n,i (0)

Original: =
Revised: =

Original: σ m,i (0) =
Revised: σ m,i (0) =

Original: N √ σ 0 , the above dynamics are the same for all singular values and we get ∀ n, m, i : σ n,i ( t )
Revised: N √ σ 0 , the above dynamics are the same for all singular values and we get ∀ n, m, i : σ n,i ( t )

Original: = σ m,i ( t ) =
Revised: = σ m,i ( t ) =

Original: N (cid:112) σ i ( t ) .
Revised: N (cid:112) σ i ( t ) .

Original: We may now use this to calculate the dynamics of the singular value of W , since they are the product the the singular values of all W n matrices.
Revised: We may now use this to calculate the dynamics of the singular value of W , since they are the product the the singular values of all W n matrices.

Original: Denoting σ − n,i = (cid:81)
Revised: Denoting σ − n,i = (cid:81)

Original: k (cid:54) = n σ k,i and using the chain rule: ˙
Revised: k (cid:54) = n σ k,i and using the chain rule: ˙

Original: [Equation]
Revised: [Equation]

Original: E. E MPIRICAL E XAMINATION
Revised: E. E MPIRICAL E XAMINATION

Original: Our analytical results are only applicable for the population loss over Gaussian inputs.
Revised: Our analytical results are only applicable for the population loss over Gaussian inputs.

Original: These conditions are far from the ones used in practice and studied in Arora et al.
Revised: These conditions are far from the ones used in practice and studied in Arora et al.

Original: (2019), where the problem is over-determined and the weights are drawn from a Gaussian distribution with a small variance.
Revised: (2019), where the problem is over-determined and the weights are drawn from a Gaussian distribution with a small variance.

Original: To show our conclusions regarding incremental learning extend qualitatively to more natural settings, we empirically examine the deep matrix sensing model in this natural setting for different depths and initialization scales as seen in ﬁgure 4.
Revised: To show our conclusions regarding incremental learning extend qualitatively to more natural settings, we empirically examine the deep matrix sensing model in this natural setting for different depths and initialization scales as seen in ﬁgure 4.

Original: Notice how incremental learning is exhibited even when the number of examples is much smaller than the number of parameters in the model.
Revised: Notice how incremental learning is exhibited even when the number of examples is much smaller than the number of parameters in the model.

Original: While we can’t rely on our theory for describing the exact dynamics of the optimization for these kinds of over-determined problems, the qualitative conclusions we get from it are still applicable.
Revised: While we can’t rely on our theory for describing the exact dynamics of the optimization for these kinds of over-determined problems, the qualitative conclusions we get from it are still applicable.

Original: Another interesting phenomena we should note is that once the dataset becomes very small (the second row of the ﬁgure), we see all “currently active” singular values change at the beginning of every new phase (this is best seen in the bottom-right panel).
Revised: Another interesting phenomena we should note is that once the dataset becomes very small (the second row of the ﬁgure), we see all “currently active” singular values change at the beginning of every new phase (this is best seen in the bottom-right panel).

Original: This suggests that since there is more than one optimal solution, once we increase the current rank of our model it may ﬁnd a solution that has a different set of singular values and vectors and thus all singular values change at the beginning of a new learning phase.
Revised: This suggests that since there is more than one optimal solution, once we increase the current rank of our model it may ﬁnd a solution that has a different set of singular values and vectors and thus all singular values change at the beginning of a new learning phase.

Original: This demonstrates the importance of incremental learning for obtaining sparse solutions - once the initialization conditions and depth are such that the learning phases are distinct, gradient descent ﬁnds the optimal rank- i solution in every phase i .
Revised: This demonstrates the importance of incremental learning for obtaining sparse solutions - once the initialization conditions and depth are such that the learning phases are distinct, gradient descent ﬁnds the optimal rank- i solution in every phase i .

Original: For these dynamics to successfully recover the optimal solution at every phase, the phases need to be far enough apart from each other to allow for the singular values and vectors to change before the next phase begins.
Revised: For these dynamics to successfully recover the optimal solution at every phase, the phases need to be far enough apart from each other to allow for the singular values and vectors to change before the next phase begins.

Original: F I NCREMENTAL L EARNING IN Q UADRATIC N ETWORKS F. P ROOF OF T HEOREM
Revised: F I NCREMENTAL L EARNING IN Q UADRATIC N ETWORKS F. P ROOF OF T HEOREM

Original: Theorem.
Revised: Theorem.

Original: Minimizing the quadratic network described and initialized as in equation 16 with gradient ﬂow over the variance loss deﬁned in equation 2 with Gaussian inputs leads to the following dynamical equations:
Revised: Minimizing the quadratic network described and initialized as in (7) with gradient ﬂow over the variance loss deﬁned in (2) with Gaussian inputs leads to the following dynamical equations:

Original: [Equation]
Revised: [Equation]

Original: Where σ i and σ ∗ i are the i th singular values of W and W ∗ , respectively, corresponding to the same singular vectors.
Revised: Where σ i and σ ∗ i are the i th singular values of W and W ∗ , respectively, corresponding to the same singular vectors.

Original: Proof.
Revised: Proof.

Original: Our proof will follow similar lines as the analysis of the deep matrix sensing model.
Revised: Our proof will follow similar lines as the analysis of the deep matrix sensing model.

Original: Taking the expectation of the variance loss over Gaussian inputs for our model gives us:
Revised: Taking the expectation of the variance loss over Gaussian inputs for our model gives us:

Original: [Equation]
Revised: [Equation]

Original: Following the gradient ﬂow dynamics over W leads to the following differential equation:
Revised: Following the gradient ﬂow dynamics over W leads to the following differential equation:

Original: [Equation]
Revised: [Equation]

Original: We can now calculate the gradient ﬂow dynamics of W T W using the chain rule:
Revised: We can now calculate the gradient ﬂow dynamics of W T W using the chain rule:

Original: [Equation]
Revised: [Equation]

Original: Now, under our initialization W T 0 W 0 = σ 0 I , we get that W T W and W T ∗ W ∗ are simultaneously diagonalizable at initialization by some matrix U , such that the following is true for diagonal D and
Revised: Now, under our initialization W T 0 W 0 = σ 0 I , we get that W T W and W T ∗ W ∗ are simultaneously diagonalizable at initialization by some matrix U , such that the following is true for diagonal D and

Original: D ∗ :
Revised: D ∗ :

Original: [Equation]
Revised: [Equation]

Original: Multiplying equation equation 49 by U and U T gives us the following dynamics for the singular values of W T W :
Revised: Multiplying equation (10) by U and U T gives us the following dynamics for the singular values of W T W :

Original: [Equation]
Revised: [Equation]

Original: These matrices are diagonal at initialization, and remain diagonal throughout the dynamics (the offdiagonal elements are static according to these equations).
Revised: These matrices are diagonal at initialization, and remain diagonal throughout the dynamics (the offdiagonal elements are static according to these equations).

Original: We may now look at the dynamics of a single diagonal element, noticing it is equivalent to the depth-2 toy model:
Revised: We may now look at the dynamics of a single diagonal element, noticing it is equivalent to the depth-2 toy model:

Original: [Equation]
Revised: [Equation]

Original: F. D ISCUSSION OF THE V ARIANCE L OSS
Revised: F. D ISCUSSION OF THE V ARIANCE L OSS

Original: It may seem that the variance loss is an unnatural loss function to analyze, since it isn’t used in practice.
Revised: It may seem that the variance loss is an unnatural loss function to analyze, since it isn’t used in practice.

Original: While this is true, we will show how the dynamics of this loss function are an approximation of the square loss dynamics.
Revised: While this is true, we will show how the dynamics of this loss function are an approximation of the square loss dynamics.

Original: We begin by describing the dynamics of both losses, showing how incremental learning can’t take place for quadratic networks as deﬁned over the squared loss.
Revised: We begin by describing the dynamics of both losses, showing how incremental learning can’t take place for quadratic networks as deﬁned over the squared loss.

Original: Then, we show how adding a global bias to the quadratic network leads to similar dynamics for small initialization scales
Revised: Then, we show how adding a global bias to the quadratic network leads to similar dynamics for small initialization scales.

Original: . F.2. D YNAMICAL D ERIVATION
Revised: F.2. D YNAMICAL D ERIVATION

Original: in the previous section, we derive the differential equations for the singular values of W T W under the variance loss:
Revised: in the previous section, we derive the differential equations for the singular values of W T W under the variance loss:

Original: [Equation]
Revised: [Equation]

Original: We will now derive similar equations for the squared loss.
Revised: We will now derive similar equations for the squared loss.

Original: The scaled squared loss in expectation over the Gaussian inputs is:
Revised: The scaled squared loss in expectation over the Gaussian inputs is:

Original: [Equation]
Revised: [Equation]

Original: Deﬁning ∆ = W T ∗ W ∗ − W T W for brevity, the differential equations for W become:
Revised: Deﬁning ∆ = W T ∗ W ∗ − W T W for brevity, the differential equations for W become:

Original: [Equation]
Revised: [Equation]

Original: Calculating the dynamics of W T W after noting that it is simultaneously diagonalizable with W T ∗ W ∗ (as in the derivation for the variance loss) leads to the following differential equations for the singular values of W T W :
Revised: Calculating the dynamics of W T W after noting that it is simultaneously diagonalizable with W T ∗ W ∗ (as in the derivation for the variance loss) leads to the following differential equations for the singular values of W T W :

Original: [Equation]
Revised: [Equation]

Original: We see that the equations are now coupled and so we cannot solve them analytically.
Revised: We see that the equations are now coupled and so we cannot solve them analytically.

Original: Another issue is that for our initialization, all singular values have very similar dynamics at initialization due to the coupling.
Revised: Another issue is that for our initialization, all singular values have very similar dynamics at initialization due to the coupling.

Original: For example, values corresponding to small optimal singular values grow much faster than in the variance loss dynamics, due to the effect the large optimal singular values have on them.
Revised: For example, values corresponding to small optimal singular values grow much faster than in the variance loss dynamics, due to the effect the large optimal singular values have on them.

Original: We see from these equations that we shouldn’t expect quadratic networks optimized over the squared loss to exhibit incremental learning behavior.
Revised: We see from these equations that we shouldn’t expect quadratic networks optimized over the squared loss to exhibit incremental learning behavior.

Original: We next show how adding a global bias to our model can help.
Revised: We next show how adding a global bias to our model can help.

Original: F.2. V ARIANCE L OSS A PPROXIMATES THE S QUARED L OSS
Revised: F.2. V ARIANCE L OSS A PPROXIMATES THE S QUARED L OSS

Original: To see how the variance loss can have dynamics resembling those of the squared loss, we will add a global (trainable) bias to our model.
Revised: To see how the variance loss can have dynamics resembling those of the squared loss, we will add a global (trainable) bias to our model.

Original: This means our model is now parameterized by W ∈ R d × d and a scalar b ∈ R :
Revised: This means our model is now parameterized by W ∈ R d × d and a scalar b ∈ R :

Original: [Equation]
Revised: [Equation]

Original: We may now analyze gradient ﬂow of the squared loss.
Revised: We may now analyze gradient ﬂow of the squared loss.

Original: Following the same methods as before, this leads us to the following differential equations:
Revised: Following the same methods as before, this leads us to the following differential equations:

Original: [Equation]
Revised: [Equation]

Original: Notice how, if b is at it’s optimum at a given time ( b = (cid:80) j ( σ ∗ j − σ j ) + b ∗ ), the dynamics of σ i align with those of the variance loss.
Revised: Notice how, if b is at it’s optimum at a given time ( b = (cid:80) j ( σ ∗ j − σ j ) + b ∗ ), the dynamics of σ i align with those of the variance loss.

Original: Alternatively, when b = b ∗ , we recover the dynamics of the squared loss without the global bias term.
Revised: Alternatively, when b = b ∗ , we recover the dynamics of the squared loss without the global bias term.

Original: To convince ourselves that the dynamics of this model resemble those of the variance loss, we would need to explain why the global bias is at it’s optimum “most of the time”, such that the singular values don’t change much during the times when it is not at it’s optimum.
Revised: To convince ourselves that the dynamics of this model resemble those of the variance loss, we would need to explain why the global bias is at it’s optimum “most of the time”, such that the singular values don’t change much during the times when it is not at it’s optimum.

Original: Observing the differential equations for b and for σ i , we see they are similar (if we ignore the σ ∗ i − σ i value which doesn’t change the order of magnitude of the entire expression when there haven’t been many learning phases yet).
Revised: Observing the differential equations for b and for σ i , we see they are similar (if we ignore the σ ∗ i − σ i value which doesn’t change the order of magnitude of the entire expression when there haven’t been many learning phases yet).

Original: The only difference being a multiplication by σ i .
Revised: The only difference being a multiplication by σ i .

Original: This means that we may informally write:
Revised: This means that we may informally write:

Original: [Equation]
Revised: [Equation]

Original: Since at initialization and until the learning phase of σ i takes place, we have σ i ( t ) (cid:28) 1 , we see that the global bias optimizes much faster than the singular values for which the learning phase hasn’t begun yet.
Revised: Since at initialization and until the learning phase of σ i takes place, we have σ i ( t ) (cid:28) 1 , we see that the global bias optimizes much faster than the singular values for which the learning phase hasn’t begun yet.

Original: This means these singular values will remain small during the times in which
Revised: This means these singular values will remain small during the times in which

Original: Note how without the bias, the singular values are learned together and there is over-shooting of the optimal singular value caused by the coupling of the dynamics of the singular values.
Revised: Note how without the bias, the singular values are learned together and there is over-shooting of the optimal singular value caused by the coupling of the dynamics of the singular values.

Original: For the small datasets, we see that the model with no bias reaches a solution with a larger rank.
Revised: For the small datasets, we see that the model with no bias reaches a solution with a larger rank.

Original: Once a global bias is introduced, the dynamics become more incremental as in the analysis of the variance loss.
Revised: Once a global bias is introduced, the dynamics become more incremental as in the analysis of the variance loss.

Original: Note that in this case the solution obtained for the small dataset is the optimal low-ranksolution. the bias isn’t optimal, and so incremental learning can still take place (assuming a small enough initialization).
Revised: that in this case the solution obtained for the small dataset is the optimal low-rank the bias isn’t optimal, and so incremental learning can still take place (assuming a small enough initialization).

Original: Under these considerations, we say that the dynamics of the squared loss for a quadratic network with an added global bias resemble the idealized dynamics of the variance loss for a depth-2 linear model which we analyze formally in the paper.
Revised: Under these considerations, we say that the dynamics of the squared loss for a quadratic network with an added global bias resemble the idealized dynamics of the variance loss for a depth-2 linear model which we analyze formally in the paper.

Original: In ﬁgure 5 we experimentally show how adding a bias to a quadratic network does lead to incremental learning similar to the depth-2 toy model.
Revised: In ﬁgure 5 we experimentally show how adding a bias to a quadratic network does lead to incremental learning similar to the depth-2 toy model.

Original: G I NCREMENTAL L EARNING IN C LASSIFICATION G. D IAGONAL N ETWORKS
Revised: G I NCREMENTAL L EARNING IN C LASSIFICATION G. D IAGONAL N ETWORKS

Original: In section 4.3 we viewed our toy model as a special case of the deep diagonal networks described in Gunasekar et al.
Revised: In section 4.3 we viewed our toy model as a special case of the deep diagonal networks described in Gunasekar et al.

Original: (2018), expected to be biased towards sparse solutions.
Revised: (2018), expected to be biased towards sparse solutions.

Original: Figure 6 shows the dynamics of the largest values of σ for different depths of the model.
Revised: Figure 6 shows the dynamics of the largest values of σ for different depths of the model.

Original: We see that the same type of incremental learning we saw in earlier models exists here as well - the features are learned one by one in deeper models, resulting in a sparse solution.
Revised: We see that the same type of incremental learning we saw in earlier models exists here as well - the features are learned one by one in deeper models, resulting in a sparse solution.

Original: The leftmost panel shows how the initialization scale plays a role here as well, with the solution being more sparse when the initialization is small.
Revised: The leftmost panel shows how the initialization scale plays a role here as well, with the solution being more sparse when the initialization is small.

Original: We should note that these results do not defy the results of Gunasekar et al.
Revised: We should note that these results do not defy the results of Gunasekar et al.

Original: (2018) (from which we would expect the initialization not to matter), since their results deal with the solution at t → ∞ .
Revised: (2018) (from which we would expect the initialization not to matter), since their results deal with the solution at t → ∞ .

Original: G. C ONVOLUTIONAL N ETWORKS - P RELIMINARIES
Revised: G. C ONVOLUTIONAL N ETWORKS - P RELIMINARIES

Original: The linear circular-convolutional network of Gunasekar et al.
Revised: The linear circular-convolutional network of Gunasekar et al.

Original: (2018) deals with one-dimensional convolutions with the same number of outputs as inputs, such that the mapping from one hidden layer to the next is parameterized by w n and deﬁned to be:
Revised: (2018) deals with one-dimensional convolutions with the same number of outputs as inputs, such that the mapping from one hidden layer to the next is parameterized by w n and deﬁned to be:

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal layer is a fully connected layer parameterized by w N ∈ R d , such that the ﬁnal model can be written in the following way:
Revised: The ﬁnal layer is a fully connected layer parameterized by w N ∈ R d , such that the ﬁnal model can be written in the following way:

Original: [Equation]
Revised: [Equation]

Original: Lemma 3 from Gunasekar et al.
Revised: Lemma 3 from Gunasekar et al.

Original: (2018) shows how we can relate the Fourier coefﬁcients of the weight vectors to the Fourier coefﬁcients of the linear model induced by the model:
Revised: (2018) shows how we can relate the Fourier coefﬁcients of the weight vectors to the Fourier coefﬁcients of the linear model induced by the model:

Original: Lemma.
Revised: Lemma.

Original: For the circular-convolutional model as in equation 56:
Revised: For the circular-convolutional model as in (11) :

Original: [Equation]
Revised: [Equation]

Original: where for n = 1 , 2 , ..., N, ˆ w n ∈ C
Revised: where for n = 1 , 2 , ..., N, ˆ w n ∈ C

Original: d are the Fourier coefﬁcients of the parameters w n ∈ R d .
Revised: d are the Fourier coefﬁcients of the parameters w n ∈ R d .

Original: This lemma connects the convolutional network to the diagonal network, and thus we should expect to see the same incremental learning of the values of the diagonal network exhibited by the Fourier coefﬁcients of the convolutional network.
Revised: This lemma connects the convolutional network to the diagonal network, and thus we should expect to see the same incremental learning of the values of the diagonal network exhibited by the Fourier coefﬁcients of the convolutional network.

Original: G.
Revised: G.

Original: C ONVOLUTIONAL N ETWORKS - E MPIRICAL E XAMINATION
Revised: C ONVOLUTIONAL N ETWORKS - E MPIRICAL E XAMINATION

Original: In ﬁgure 7 we see the same plots as in ﬁgure 6 but for the Fourier coefﬁcients of the convolutional model.
Revised: In ﬁgure 7 we see the same plots as in ﬁgure 6 but for the Fourier coefﬁcients of the convolutional model.

Original: We see that even when the model is far from the toy parameterization (there is no weight sharing and the initialization is with random Gaussian weights), incremental learning is still clearly seen in the dynamics of the model.
Revised: We see that even when the model is far from the toy parameterization (there is no weight sharing and the initialization is with random Gaussian weights), incremental learning is still clearly seen in the dynamics of the model.

Original: We see how the inherent reason for the sparsity towards sparse solution found in Gunasekar et al.
Revised: We see how the inherent reason for the sparsity towards sparse solution found in Gunasekar et al.

Original: (2018) is the result of the dynamics of the model - small amplitudes are attenuated while large ones are ampliﬁed.
Revised: (2018) is the result of the dynamics of the model - small amplitudes are attenuated while large ones are ampliﬁed.