Original: On the Learning Dynamics of Deep Neural Networks
Revised: On the Learning Dynamics of Deep Neural Networks

Original: Anonymous authors Paper under double-blind review
Revised: Anonymous authors Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: While a lot of progress has been made in recent years, the dynamics of learning in deep nonlinear neural networks remain to this day largely misunderstood.
Revised: While a lot of progress has been made in recent years, the dynamics of learning in deep nonlinear neural networks remain to this day largely misunderstood.

Original: In this work, we study the case of binary classiﬁcation and prove various properties of learning in such networks under strong assumptions such as linear separability of the data.
Revised: In this work, we study the case of binary classiﬁcation and prove various properties of learning in such networks under strong assumptions such as linear separability of the data.

Original: Extending existing results from the linear case, we conﬁrm empirical observations by proving that the classiﬁcation error also follows a sigmoidal shape in nonlinear architectures.
Revised: Extending existing results from the linear case, we conﬁrm empirical observations by proving that the classiﬁcation error also follows a sigmoidal shape in nonlinear architectures.

Original: We show that given proper initialization, learning expounds parallel independent modes and that certain regions of parameter space might lead to failed training.
Revised: We show that given proper initialization, learning expounds parallel independent modes and that certain regions of parameter space might lead to failed training.

Original: We also demonstrate that input norm and features’ frequency in the dataset lead to distinct convergence speeds which might shed some light on the generalization capabilities of deep neural networks.
Revised: We also demonstrate that input norm and features’ frequency in the dataset lead to distinct convergence speeds which might shed some light on the generalization capabilities of deep neural networks.

Original: We provide a comparison between the dynamics of learning with cross-entropy and hinge losses, which could prove useful to understand recent progress in the training of generative adversarial networks.
Revised: We provide a comparison between the dynamics of learning with cross-entropy and hinge losses, which could prove useful to understand recent progress in the training of generative adversarial networks.

Original: Finally, we identify a phenomenon that we baptize gradient starvation where the most frequent features in a dataset prevent the learning of other less frequent but equally informative features.
Revised: Finally, we identify a phenomenon that we baptize gradient starvation where the most frequent features in a dataset prevent the learning of other less frequent but equally informative features.

Original:  Introduction
Revised:  Introduction

Original: Due to extremely complex interactions between millions of parameters, nonlinear activation functions and optimization techniques, the dynamics of learning observed in deep neural networks remain much of a mystery to this day.
Revised: Due to extremely complex interactions between millions of parameters, nonlinear activation functions and optimization techniques, the dynamics of learning observed in deep neural networks remain much of a mystery to this day.

Original: What principles govern the evolution of the neural network weights?
Revised: What principles govern the evolution of the neural network weights?

Original: Why does the training error evolve as it does?
Revised: Why does the training error evolve as it does?

Original: How do data and optimization techniques like stochastic gradient descent interact?
Revised: How do data and optimization techniques like stochastic gradient descent interact?

Original: Where does the implicit regularization of deep neural networks trained with stochastic gradient descent come from?
Revised: Where does the implicit regularization of deep neural networks trained with stochastic gradient descent come from?

Original: Shedding some light on those questions would make training neural networks more understandable and potentially pave the way to better techniques.
Revised: Shedding some light on those questions would make training neural networks more understandable and potentially pave the way to better techniques.

Original: It is commonly accepted that learning is composed of alternating phases: plateaus where the error remains fairly constant and periods of fast improvement where a lot of progress is made over the course of few epochs (Saxe et al., 2013a).
Revised: It is commonly accepted that learning is composed of alternating phases: plateaus where the error remains fairly constant and periods of fast improvement where a lot of progress is made over the course of few epochs (Saxe et al., 2013a).

Original: Theoretic explanations of that phenomenon exist in the case of regression on linear neural networks (Saxe, 2015) but extensions to the nonlinear case (Heskes & Kappen, 1993; Raghu et al., 2017; Arora et al., 2018) fail to provide analytical solutions.
Revised: Theoretic explanations of that phenomenon exist in the case of regression on linear neural networks (Saxe, 2015) but extensions to the nonlinear case (Heskes & Kappen, 1993; Raghu et al., 2017; Arora et al., 2018) fail to provide analytical solutions.

Original: It has been observed in countless experiments that deep networks present strong generalization abilities.
Revised: It has been observed in countless experiments that deep networks present strong generalization abilities.

Original: Those abilities are however diﬃcult to ground in solid theoretical foundations.
Revised: Those abilities are however diﬃcult to ground in solid theoretical foundations.

Original: The fact that deep network have millions of parameters – a number sometimes orders of magnitude larger than the dataset size – contradicts the expectations set by classic statistical learning theory on the necessity of regularizers (Vapnik, 1998; Poggio et al., 2004).
Revised: The fact that deep network have millions of parameters – a number sometimes orders of magnitude larger than the dataset size – contradicts the expectations set by classic statistical learning theory on the necessity of regularizers (Vapnik, 1998; Poggio et al., 2004).

Original: This observation drove Zhang et al.
Revised: This observation drove Zhang et al.

Original: (2016) to suggest the existence of an implicit regularization happening during the training of deep neural networks.
Revised: (2016) to suggest the existence of an implicit regularization happening during the training of deep neural networks.

Original: Advani & Saxe (2017) show that the dynamics of gradient descent can protect against overﬁtting in large networks.
Revised: Advani & Saxe (2017) show that the dynamics of gradient descent can protect against overﬁtting in large networks.

Original: Kleinberg et al.
Revised: Kleinberg et al.

Original: (2018) also oﬀer some explanations of the phenomenon but understanding its roots remains an open problem.
Revised: (2018) also oﬀer some explanations of the phenomenon but understanding its roots remains an open problem.

Original: In this work, we study the learning dynamics of a deep nonlinear neural network trained on a standard classiﬁcation task using two diﬀerent losses: the cross-entropy and the hinge loss.
Revised: In this work, we study the learning dynamics of a deep nonlinear neural network trained on a standard classiﬁcation task using two diﬀerent losses: the cross-entropy and the hinge loss.

Original: We mainly focus on binary classiﬁcation, some of the results and properties can however be extended to the multi-class case.
Revised: We mainly focus on binary classiﬁcation, some of the results and properties can however be extended to the multi-class case.

Original: The questions we address in Sections 3, 4 and 5 respectively can be summarized as follows:
Revised: The questions we address in Sections 3, 4 and 5 respectively can be summarized as follows:

Original: How does the conﬁdence of a classiﬁer evolve throughout learning?
Revised: How does the conﬁdence of a classiﬁer evolve throughout learning?

Original: How does the loss used during training impact its dynamics?
Revised: How does the loss used during training impact its dynamics?

Original: Which properties of the features present in a dataset impact learning, and how?
Revised: Which properties of the features present in a dataset impact learning, and how?

Original: Independent mode learning We show that, similarly to the case of linear networks and under certain initial conditions, learning happens independently between diﬀerent classes, i.e. classes induce a partition of the network activations, corresponding to orthogonal modes of the data.
Revised: Independent mode learning We show that, similarly to the case of linear networks and under certain initial conditions, learning happens independently between diﬀerent classes, i.e. classes induce a partition of the network activations, corresponding to orthogonal modes of the data.

Original: Learning dynamics We prove that in accordance to experimental ﬁndings, the hidden activations and the classiﬁcation error of the network show a sigmoidal shape with slow learning at the beginning followed by fast saturation of the curve.
Revised: Learning dynamics We prove that in accordance to experimental ﬁndings, the hidden activations and the classiﬁcation error of the network show a sigmoidal shape with slow learning at the beginning followed by fast saturation of the curve.

Original: We also characterize a region in the initialization space where learning is frozen or eventually dies out.
Revised: We also characterize a region in the initialization space where learning is frozen or eventually dies out.

Original: Hinge loss We study how using the hinge loss impacts learning and quantitatively compare it to the classic cross-entropy loss.
Revised: Hinge loss We study how using the hinge loss impacts learning and quantitatively compare it to the classic cross-entropy loss.

Original: We show that the hinge loss allows one to solve a classiﬁcation task much faster, by providing strong gradients no matter how close to convergence the neural network is.
Revised: We show that the hinge loss allows one to solve a classiﬁcation task much faster, by providing strong gradients no matter how close to convergence the neural network is.

Original: Gradient starvation Finally, we identify a phenomenon that we call gradient starvation where the most frequent features present in the dataset starve the learning of other very informative but less frequent features.
Revised: Gradient starvation Finally, we identify a phenomenon that we call gradient starvation where the most frequent features present in the dataset starve the learning of other very informative but less frequent features.

Original: Gradient starvation occurs naturally when training a neural network with gradient descent and might be part of the explanation as to why neural networks generalize so well.
Revised: Gradient starvation occurs naturally when training a neural network with gradient descent and might be part of the explanation as to why neural networks generalize so well.

Original: They intrinsically implement a variant of Occam’s razor (Ariew, 1976): the simplest explanation is the one they converge to ﬁrst .
Revised: They intrinsically implement a variant of Occam’s razor (Ariew, 1976): the simplest explanation is the one they converge to ﬁrst .

Original:  Setup and notations
Revised:  Setup and notations

Original: We are interested in a simple binary classiﬁcation task, solved by training a deep neural network with gradient descent.
Revised: We are interested in a simple binary classiﬁcation task, solved by training a deep neural network with gradient descent.

Original: This simple setup encompasses for instance the training of generative adversarial networks discriminators.
Revised: This simple setup encompasses for instance the training of generative adversarial networks discriminators.

Original: Some of our results extend to multi-class classiﬁcation, but, for the sake of conciseness, that case isonly treated in the Appendix.
Revised: Some of our results extend to multi-class classiﬁcation, but, for the sake of conciseness, that case is treated in Appendix A.

Original: We let D =
Revised: We let D =

Original: ( x i , l i ) } 1 ≤ i ≤ n ⊂ R d × { 1 , 2 } denote our dataset of vectors and labels.
Revised: ( x i , l i ) } 1 ≤ i ≤ n ⊂ R d × { 1 , 2 } denote our dataset of vectors and labels.

Original: The classiﬁer we consider is a simple neural network with one hidden layer of h neurons and a ReLU non-linearity
Revised: The classiﬁer we consider is a simple neural network with one hidden layer of h neurons and a ReLU non-linearity

Original: (see Fig. 1).
Revised: (see Fig. 1).

Original: The output of the network is passed through a function denoted o which is either the sigmoid σ in the binary cross-entropy case or the identity in the hinge loss case.
Revised: The output of the network is passed through a function denoted o which is either the sigmoid σ in the binary cross-entropy case or the identity in the hinge loss case.

Original: The full function can be written as
Revised: The full function can be written as

Original: [Equation]
Revised: [Equation]

Original: where W t is an h × d matrix and Z t a vector of length h .
Revised: where W t is an h × d matrix and Z t a vector of length h .

Original: In the C -class case, Z t is instead a C × h matrix and o the softmax function.
Revised: In the C -class case, Z t is instead a C × h matrix and o the softmax function.

Original: We will see below an extension to deeper networks. W t and Z t are the parameters of the neural network.
Revised: An extension to deeper networks can be found in Appendix B. W t and Z t are the parameters of the neural network.

Original: The subscript + (resp.
Revised: The subscript + (resp.

Original: t ) denotes the positive part of a real number (resp.
Revised: t ) denotes the positive part of a real number (resp.

Original: the state of the element at time step t of training).
Revised: the state of the element at time step t of training).

Original: The superscript T stands for the transpose operation.
Revised: The superscript T stands for the transpose operation.

Original: In the case of cross-entropy, P t ( x ) represents the probability that x belongs to class 1, for the hinge loss, P t ( x ) is trained to reach { 1 , − 1 } for classes 1 and 2 .
Revised: In the case of cross-entropy, P t ( x ) represents the probability that x belongs to class 1, for the hinge loss, P t ( x ) is trained to reach { 1 , − 1 } for classes 1 and 2 .

Original: For a given class k , we let D k denote the set of vectors belonging to it.
Revised: For a given class k , we let D k denote the set of vectors belonging to it.

Original: We make the assumption (H1)
Revised: We make the assumption (H1)

Original: For any x, x (cid:48) ∈ D k , x T x (cid:48) > 0 .
Revised: For any x, x (cid:48) ∈ D k , x T x (cid:48) > 0 .

Original: For any x ∈ D 1 and x (cid:48) ∈ D 2 , x T x (cid:48) ≤ 0 .
Revised: For any x ∈ D 1 and x (cid:48) ∈ D 2 , x T x (cid:48) ≤ 0 .

Original: It implies linear separability of the data, an assumption often necessary in theoretical studies (Soudry et al., 2017; Liao & Couillet, 2018;
Revised: It implies linear separability of the data, an assumption often necessary in theoretical studies (Soudry et al., 2017; Liao & Couillet, 2018;

Original: Nacson et al., 2018; Xu et al., 2018) and the positioning of the origin between the two sets.
Revised: Nacson et al., 2018; Xu et al., 2018) and the positioning of the origin between the two sets.

Original: It is a very strong assumption which admittedly bypasses a large part of the deep learning dynamics.
Revised: It is a very strong assumption which admittedly bypasses a large part of the deep learning dynamics.

Original: Nevertheless, it allows the discovery of interesting properties and helps develop intuitions about behaviors observed in more general settings.
Revised: Nevertheless, it allows the discovery of interesting properties and helps develop intuitions about behaviors observed in more general settings.

Original:  Learning dynamics for binary cross-entropy
Revised:  Learning dynamics for binary cross-entropy

Original: In this section, we focus on the case of the binary cross-entropy loss
Revised: In this section, we focus on the case of the binary cross-entropy loss

Revised: L

Original: [Equation]
Revised: [Equation]

Original: t ( W t x )+ )) , and train our network using stochastic gradient descent to minimize L BCE .
Revised: [Equation] ) , and train our network using stochastic gradient descent to minimize L BCE .

Original: 3.1 Independent modes of learning
Revised: 3.1 Independent modes of learning

Original: Our ﬁrst lemma states that over the course of training and under suitable initialization, the active neurons of the hidden layer remain the same for each datapoint, and the coordinates of Z t remain of the same sign.
Revised: Our ﬁrst lemma states that over the course of training and under suitable initialization, the active neurons of the hidden layer remain the same for each datapoint, and the coordinates of Z t remain of the same sign.

Original: To prove it, we let w it denote the i -th row of W t and make the additional assumptions: there exists a partition {I 1 , I 2 } of { 1 , . .
Revised: To prove it, we let w it denote the i -th row of W t and make the additional assumptions: there exists a partition {I 1 , I 2 } of { 1 , . .

Original: , h } such that with k ∈ { 1 , 2 }
Revised: , h } such that with k ∈ { 1 , 2 }

Original: (H2)
Revised: (H2)

Original: For any

Original: i

Original: ∈

Original: I k , x

Original: ∈

Original:  D k and x (cid:48) / ∈ D k , w i  x >  and w i  x (cid:48) ≤ 0 .
Revised: For any i ∈ I k , x ∈ D k and x (cid:48) / ∈ D k , w i 0 x > 0 and w i 0 x (cid:48) ≤ 0 .

Original: (H3)
Revised: (H3)

Original: The i -th coordinate of Z 0 is positive if i ∈ I 1 , negative otherwise.
Revised: The i -th coordinate of Z 0 is positive if i ∈ I 1 , negative otherwise.

Original: Assumption (H2) states that at the beginning of training, data points from diﬀerent classes do not activate the same neurons.
Revised: Assumption (H2) states that at the beginning of training, data points from diﬀerent classes do not activate the same neurons.

Original: It is an analogue to the orthogonal initialization used in Saxe et al.
Revised: It is an analogue to the orthogonal initialization used in Saxe et al.

Original: (2013b).
Revised: (2013b).

Original: In Appendix A.6, we show that relaxing it hints towards an extended period of slow learning in the early stages of training.
Revised: In Appendix A.8, we show that relaxing it hints towards an extended period of slow learning in the early stages of training.

Original: (H3) is introduced for Lemma 3.1 and Theorem 3.2 but will be relaxed later.
Revised: (H3) is introduced for Lemma 3.1 and Theorem 3.2 but will be relaxed later.

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: For any k ∈ { 1 , 2 } , x ∈ D k and t ≥ 0 , the only non-negative elements of W t x are the ones with an index i ∈
Revised: For any k ∈ { 1 , 2 } , x ∈ D k and t ≥ 0 , the only non-negative elements of W t x are the ones with an index i ∈

Original: I k .
Revised: I k .

Original: The signs of the coordinates of Z t remain the same throughout training.
Revised: The signs of the coordinates of Z t remain the same throughout training.

Original: This lemma proves that updates to the parameters of our network are fully decoupled from one class to the other.
Revised: This lemma proves that updates to the parameters of our network are fully decoupled from one class to the other.

Original: An update for a data point in D k will only inﬂuence the corresponding active rows and elements of W t and Z t .
Revised: An update for a data point in D k will only inﬂuence the corresponding active rows and elements of W t and Z t .

Original: This "independent mode learning" is an equivalent of the results by Saxe et al.
Revised: This "independent mode learning" is an equivalent of the results by Saxe et al.

Original: (2013b) in a non-linear network trained on the cross entropy loss.
Revised: (2013b) in a non-linear network trained on the cross entropy loss.

Original: The proof of the lemma and its extension to N − 1 hidden layers and multi-class classiﬁcation can be found in Appendix A.
Revised: The proof of the lemma and its extension to N − 1 hidden layers and multi-class classiﬁcation can be found in Appendix A.

Original: 3.2 Learning dynamics
Revised: 3.2 Learning dynamics

Original: We are now interested in the actual dynamics of learning, and move from discrete updates to continuous ones by considering an inﬁnitesimal learning rate α (Heskes & Kappen, 1993).
Revised: We are now interested in the actual dynamics of learning, and move from discrete updates to continuous ones by considering an inﬁnitesimal learning rate α (Heskes & Kappen, 1993).

Original: Lemma 3.1 can easily be extended to this setting.
Revised: Lemma 3.1 can easily be extended to this setting.

Original: For simplicity we assume h = 2 , but similar results hold for arbitrary h (see Appendix A.4).
Revised: For simplicity we assume h = 2 , but similar results hold for arbitrary h (see Appendix A.4).

Original: For the moment, we maintain the assumptions (H1-3).
Revised: For the moment, we maintain the assumptions (H1-3).

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Assuming that each class k contains the same vector  repeated | D k | times, then the output of the classiﬁer on that class converges at a speed proportional to both | D k | / | D |and to the norm of the vector it contains.
Revised: Assuming that each class k contains the same vector x k repeated | D k | times, then the output of the classiﬁer on D k veriﬁes (with p k = | D k | / | D | the fraction of D belonging to D k ): [Equation] where u is deﬁned below.

Original: The shape of the learning dynamics is shown on Fig. 
Revised: The classiﬁcation curves are sigmoidal and can be found on Fig. 2 Right.

Original: Proof.
Revised: Proof.

Original: To simplify the notations, we arbitrarily assume that I 1 = { 1 } and we write w t and z t the row and element modiﬁed by an update made using x ∈ D 1 (the case of D 2 can be treated symmetrically).
Revised: To simplify the notations, we arbitrarily assume that I 1 = { 1 } and we write w t and z t the row and element modiﬁed by an update made using x ∈ D 1 (the case of D 2 can be treated symmetrically).

Original: By the independenceproven above, we know that w t and z t are only aﬀected by updates from D 1 .
Revised: By the independence above, we know that w t and z t are only aﬀected by updates from D 1 .

Original: This greatly simpliﬁes our evolution equations to: w (cid:48) t
Revised: This greatly simpliﬁes our evolution equations to: w (cid:48) t

Original: = δ f ( x ) z t x T and z (cid:48) t
Revised: = δ f ( x ) z t x T and z (cid:48) t

Original: = δ f ( x ) w t x where δ f ( x ) is the gradient of the loss with respect to the pre-sigmoid output of the network u t ( x ) : δ f ( x ) = 1 { k =1 } − σ ( u t ( x )) and the prime indicates a time derivative.
Revised: = δ f ( x ) w t x where δ f ( x ) is the gradient of the loss with respect to the pre-sigmoid output of the network u t ( x ) : δ f ( x ) = 1 { k =1 } − σ ( u t ( x )) and the prime indicates a time derivative.

Original: We let y t =
Revised: We let y t =

Original: w t x , which gives
Revised: w t x , which gives

Original: [Equation]
Revised: [Equation]

Original: Writing (cid:107) x (cid:107) 

Revised: Writing (cid:107) x (cid:107)

Original: = x T x , we see that the quantity y 2 t − (cid:107) x (cid:107) 2 z 2 t is an invariant of the problem, so its solutions live on hyperbolas of equation y 2 − (cid:107) x (cid:107) 

Revised: = x T x , we see that the quantity y 2 t − (cid:107) x (cid:107) 2 z 2 t is an invariant of the problem, so its solutions live on hyperbolas of equation y 2 − (cid:107) x (cid:107)

Original: z 2 = ± c with c := | y 20 − (cid:107) x (cid:107) 2 z 20 | .
Revised: z 2 = ± c with c := | y 20 − (cid:107) x (cid:107) 2 z 20 | .

Original: We only treat the case of a degenerate hyperbola c = 0 i.e. y 

Revised: We only treat the case of a degenerate hyperbola c = 0 i.e. y

Original: = (cid:107) x (cid:107) 2 z 20 , and refer the interested reader to Appendix A.3 for the full derivation.
Revised: = (cid:107) x (cid:107) 2 z 20 , and refer the interested reader to Appendix A.3 for the full derivation.

Original: In the case c = 0 , we have ∀ t, y t
Revised: In the case c = 0 , we have ∀ t, y t

Original: = (cid:107) x (cid:107) z t
Revised: = (cid:107) x (cid:107) z t

Original: (those quantities are both positive as x ∈ D 1 and (H2-3)).
Revised: (those quantities are both positive as x ∈ D 1 and (H2-3)).

Original: u ( t ) :
Revised: u ( t ) :

Original: =
Revised: =

Original: z t w t x = z t y t thus follows the equation  u (cid:48)
Revised: z t w t x = z t y t thus follows the equation [Figure] u [Equation]

Original: ( t ) = 2 (cid:107) x (cid:107) u
Revised: ) = [Equation] x [Equation]

Original:  ( t ) σ ( − u ( t )) .
Revised: u ( t ) σ ( − u ( t )) .

Original: One can see a clear equivalence between our evolution equation and the one obtained in Saxe et al.
Revised: One can see the equivalence between our evolution equation and Eq. (10) in Saxe et al.

Original: (2013b).
Revised: (2013b).

Original: Its analytical solution is (see Appendix A.3)
Revised: Its analytical solution is (see Appendix A.3):

Original: [Equation]
Revised: [Equation]

Original: where Ei is the exponential integral (Wiki., 2018) and < − 1> denotes the inverse function.
Revised: where Ei is the exponential integral (Wiki., 2018) and < − 1> denotes the inverse function

Original: We let ¯ u denote that function for (cid:107) x (cid:107) = 1 .
Revised: We let ¯ u denote that function for (cid:107) x (cid:107) = 1 .

Original: For x ∈ D 2 , the degeneracy assumption becomes y 0 = −(cid:107) x (cid:107) z 0 .
Revised: For x ∈ D 2 , the degeneracy assumption becomes y 0 = −(cid:107) x (cid:107) z 0 .

Original: It can be shown similarly that v ( t ) :
Revised: It can be shown similarly that v ( t ) :

Original: =
Revised: =

Original: z t y t veriﬁes the equation v (cid:48) ( t )
Revised: z t y t veriﬁes the equation v (cid:48) ( t )

Original: = 2 (cid:107) x (cid:107) v ( t ) σ ( v ( t ))[Figure] with a negative initial condition (H2-
Revised: = 2 (cid:107) x (cid:107) v ( t ) σ ( v ( t )) with a negative initial condition (H2-3).

Original: In other words,  and v follow symmetric trajectories on the /negative real line.
Revised: In other words, u and v follow symmetric trajectories on the positive/negative real line.

Original: Below, ¯ 
Revised: Below, ¯ u (resp.

Original:  ) denote those two  1 and  conditions u 0 > 0 (
Revised: ¯ v ) denote those two trajectories for (cid:107) x (cid:107) = 1 and initial conditions u 0 > 0 (resp.

Original: v 0 < 
Revised: v 0 < 0 ).

Original: Let us now write p  = | D 1 | / | D | the fraction of pointsin the dataset belonging to D 1 .
Revised: Let us now write p 1 = | D 1 | / | D | the fraction of points belonging to D 1 .

Original: Because we sample randomly from the dataset, this amounts to sampling p  (resp.
Revised: Because we sample randomly from the dataset, this amounts to sampling p 1 (resp.

Original: 1 − p  ) points from D 1 (resp.
Revised: 1 − p 1 ) points from D 1 (resp.

Original: D 2 ) for each time unit during training , i e. to rescaling the time axis by p  for D 1 and 1 − p  for D 2 .
Revised: D 2 ) for each time unit during training, i.e. to rescaling the time axis by p 1 for D 1 and 1 − p 1 for D 2 .

Original: Formally, this allows us to quantify the  performanceof the network at any time t 
Revised: This allows us to quantify the network’s performance at any time t :

Original: [Equation]
Revised: [Equation]

Original: In particular, the convergence of u ( t ) to + ∞ can be bounded using our results: convergence happens at a rate slower than log( t ) (Appendix A.3), a fact proved on its own by Soudry et al.
Revised: In particular, the convergence of u ( t ) to + ∞ can be bounded using our results: convergence happens at a rate slower than log( t ) (Appendix A.3), a fact proved on its own by Soudry et al.

Original: Interpretation Fig.
Revised: Interpretation Fig.

Original: 2 Right. shows the learning dynamics for diﬀerent values of (cid:107) x (cid:107) and c .
Revised: 2 Right. shows the learning dynamics for diﬀerent values of (cid:107) x (cid:107) and c .

Original: One common characteristic between all the curves is their sigmoidal shape.
Revised: One common characteristic between all the curves is their sigmoidal shape.

Original: Learning is slow at ﬁrst, then accelerates before saturating.
Revised: Learning is slow at ﬁrst, then accelerates before saturating.

Original: This is aligned with empirical results from the literature.
Revised: This is aligned with empirical results from the literature.

Original: We also see on e.g. the blue and yellow curves that a larger (cid:107) x (cid:107) (or similarly a larger p ) converges much faster.
Revised: We also see on e.g. the blue and yellow curves that a larger (cid:107) x (cid:107) (or similarly a larger p ) converges much faster.

Original: The eﬀect of c on the dynamics can mostly been seen at the beginning of training (for instance on the green and yellow curves).
Revised: The eﬀect of c on the dynamics can mostly been seen at the beginning of training (for instance on the green and yellow curves).

Original: It fades as convergence happens, corresponding to points of the hyperbolas getting closer to the asymptote y = (cid:107) x (cid:107) z , see Fig.
Revised: It fades as convergence happens, corresponding to points of the hyperbolas getting closer to the asymptote y = (cid:107) x (cid:107) z , see Fig.

Original: 2 Left.
Revised: 2 Left.

Original: and below for more details.
Revised: and below for more details.

Original: We can characterize the convergence speeds more quantitatively with the 
Revised: We can characterize the convergence speeds more quantitatively with the following corollary.

Original: Corollary 3.3.
Revised: Corollary 3.3.

Original: Let δ be the required accuracy on the classiﬁcation task (i.e. P t ( x ∈ D 1 ) ≥ 1 − δ for x ∈ D 1 ).
Revised: Let δ be the required accuracy on the classiﬁcation task (i.e. P t ( x ∈ D 1 ) ≥ 1 − δ for x ∈ D 1 ).

Original: Under certain assumptions, the times t ∗ 1 and t ∗ 2 required to reach that accuracy for each classiﬁer verify t

Original: [Equation]

Original: ≈

Original: [Equation] 1 (cid:107) (cid:107) x 2 (cid:107) p [Equation] p  where (cid:107) x k (cid:107) is the norm of vector 
Revised: Under certain assumptions, the times t ∗ 1 and t ∗ 2 required to reach that accuracy for each classiﬁer verify t ∗ 2 t ∗ 1 ≈ (cid:107) x 1 (cid:107) (cid:107) x 2 (cid:107) p 1 1 − p 1 where (cid:107) x k (cid:107) is the norm of vector (cid:107) x k (cid:107) from class k .

Original: [Equation]

Original: from class k .

Original: The proof can be found in Appendix A.5.
Revised: The proof can be found in Appendix A.5.

Original: More frequent classes and larger inputs will be classiﬁed at a given level of conﬁdence faster.
Revised: More frequent classes and larger inputs will be classiﬁed at a given level of conﬁdence faster.

Original: The class frequency observation is fairly straightforward as updates on a more frequent class occur at a higher rate.
Revised: The class frequency observation is fairly straightforward as updates on a more frequent class occur at a higher rate.

Original: As far as input sizes are considered, this can be seen as an analogous to the results from Saxe et al.
Revised: As far as input sizes are considered, this can be seen as an analogous to the results from Saxe et al.

Original: (2013b) stating that input-output correlations drive the speed of learning.
Revised: (2013b) stating that input-output correlations drive the speed of learning.

Original: Because a sigmoid is applied on the network output, its (pre-sigmoid) targets are sent to ±∞ .
Revised: Because a sigmoid is applied on the network output, its (pre-sigmoid) targets are sent to ±∞ .

Original: A larger input is more correlated with its target and converges faster.
Revised: A larger input is more correlated with its target and converges faster.

Original: On the assumptions The assumption that each class only contains one vector allows us to obtain the ﬁrst closed-form solutions of the learning dynamics for the binary cross-entropy .
Revised: On the assumptions The assumption that each class only contains one vector allows us to obtain the ﬁrst closed-form solutions of the learning dynamics for the binary cross-entropy .

Original: It can be relaxed to classes containing orthogonal datapoints (see Appendix A.8) which still remains restrictive.
Revised: It can be relaxed to classes containing orthogonal datapoints (see Appendix A.7) which still remains restrictive.

Original: A possible interpretation is the following: if one were to consider a deep neural network that has learnt two discriminative features for the two classes, applying classic SGD on those features would result in a learning rate proportional to the prominence of those two features in the original dataset, and to learning curves of that exact shape.
Revised: A possible interpretation is the following: if one were to consider a deep neural network that has learnt two discriminative features for the two classes, applying classic SGD on those features would result in a learning rate proportional to the prominence of those two features in the original dataset, and to learning curves of that exact shape.

Original: It is worth noting that such shapes are regularly observed by ML practitioners (Saxe et al., 2013b), our results reveal insights - otherwise unobtainable - into them.
Revised: It is worth noting that such shapes are regularly observed by ML practitioners (Saxe et al., 2013b), our results reveal insights - otherwise unobtainable - into them.

Original: 3.3 Phase diagram
Revised: 3.3 Phase diagram

Original: In this section, we build the phase diagram of Fig.
Revised: In this section, we build the phase diagram of Fig.

Original: 2 Left. .
Revised: 2 Left .

Original: The notations follow Theorem 3.2, in particular y t =
Revised: The notations follow Theorem 3.2, in particular y t =

Original: w t x .
Revised: w t x .

Original: So far, we have considered points in the top-right quadrant.
Revised: So far, we have considered points in the top-right quadrant.

Original: In that region, the couple ( z t , y t ) lives on a hyperbola of equation y 2 − (cid:107) x (cid:107) 2 z
Revised: In that region, the couple ( z t , y t ) lives on a hyperbola of equation y 2 − (cid:107) x (cid:107) 2 z

Original: =
Revised: =

Original: ±
Revised: ±

Original: c where c ≥ 0 .
Revised: c where c ≥ 0 .

Original: The sign in the equation is deﬁned by the position of ( z 0 , y 0 ) relative to the function
Revised: The sign in the equation is deﬁned by the position of ( z 0 , y 0 ) relative to the function

Original: y = (cid:107) x (cid:107) z
Revised: y = (cid:107) x (cid:107) z

Original: (positive if above, negative otherwise).
Revised: (positive if above, negative otherwise).

Original: If ( z 0 , y 0 ) is originally on that line, it will remain there throughout training.
Revised: If ( z 0 , y 0 ) is originally on that line, it will remain there throughout training.

Original: We now explore the rest of the parameter space by relaxing some of our assumptions.
Revised: We now explore the rest of the parameter space by relaxing some of our assumptions.

Original: We still consider a point x ∈ D 1 , the diagram for D 2 can be obtained by mirroring the z axis.
Revised: We still consider a point x ∈ D 1 , the diagram for D 2 can be obtained by mirroring the z axis.

Original: On Assumption (H2).
Revised: Assumption (H2).

Original: Let us ﬁrst consider the simple case of w 0 x = y 0 < 0 .
Revised: Let us ﬁrst consider the simple case of w 0 x = y 0 < 0 .

Original: The neuron is initially inactive because of the ReLU.
Revised: The neuron is initially inactive because of the ReLU.

Original: No updates will ever be made to w t during training.
Revised: No updates will ever be made to w t during training.

Original: This corresponds to the bottom half of the phase diagram, the parameters are frozen (also see Advani & Saxe (2017)).
Revised: This corresponds to the bottom half of the phase diagram, the parameters are frozen (also see Advani & Saxe (2017)).

Original: On Assumption (H3).
Revised: Assumption (H3).

Original: We now assume that z 0 ≤ 0 .
Revised: We now assume that z 0 ≤ 0 .

Original: In that case, a simple extension of Lemma 3. shows that learning still happens independently on each row of w t .
Revised: In that case, a simple extension of Lemma 3.1 shows that learning still happens independently on each row of w t .

Original: The outcome from Theorem 3. is still valid: the couple ( z t , y t ) lives on a hyperbola.
Revised: The outcome from Theorem 3.2 is still valid: the couple ( z t , y t ) lives on a hyperbola.

Original: It is however not guaranteed anymore that y t shall remain positive throughout training.
Revised: It is however not guaranteed anymore that y t shall remain positive throughout training.

Original: There are three possible situations (numbered 1 to 3), each represented by the corresponding point on the diagram.
Revised: There are three possible situations (numbered 1 to 3), each represented by the corresponding point on the diagram.

Original:  If y 0 = −(cid:107) x (cid:107) z 0 , then the points ( z t , y t ) are stuck in the top-left quadrant and converge to zero.
Revised: 1) If y 0 = −(cid:107) x (cid:107) z 0 , then the points ( z t , y t ) are stuck in the top-left quadrant and converge to zero.

Original: The equation veriﬁed by the logit u ( t ) is u (cid:48)
Revised: The equation veriﬁed by the logit u ( t ) is u (cid:48)

Original: ( t ) =
Revised: ( t ) =

Original: − 2 (cid:107) x (cid:107) u ( t ) σ ( − u ( t ))
Revised: − 2 (cid:107) x (cid:107) u ( t ) σ ( − u ( t ))

Original: (see Appendix A.7).
Revised: (see Appendix A.6).

Original: If y 0 > −(cid:107) x (cid:107) z  , the points ( z t , y t ) move on the hyperbola towards the top-right quadrant, at which point z t becomes positive and y t starts increasing again.
Revised: If y 0 > −(cid:107) x (cid:107) z 0 , the points ( z t , y t ) move on the hyperbola towards the top-right quadrant, at which point z t becomes positive and y t starts increasing again.

Original: If y 0 < −(cid:107) x (cid:107) z  , the points ( z t , y t ) move on the hyperbola towards the bottom-left quadrant, at which point y t becomes negative.
Revised: If y 0 < −(cid:107) x (cid:107) z 0 , the points ( z t , y t ) move on the hyperbola towards the bottom-left quadrant, at which point y t becomes negative.

Original: When that happens, the neuron dies out, learning stops.
Revised: When that happens, the neuron dies out, learning stops.

Original: [Figure]

Original: Only in the second case will the classiﬁer end up solving the task: random initialization  functions in certain parts of the ( y t , z t ) space.
Revised: Only in the second case will the classiﬁer end up solving the task: random initialization only functions in certain parts of the ( y t , z t ) space.

Original: Those ﬁndings are summarized in the phase diagram Fig.
Revised: Those ﬁndings are summarized in the phase diagram Fig.

Original: 2 Left .
Revised: Left .

Original: The red region represents the initialization where the network will not be able to solve the task.
Revised: The red region represents the initialization where the network will not be able to solve the task.

Revised: Failure modes Assumption (H2) essentially means that the network’s ﬁrst layer is able to separate the data at t = 0 .

Revised: It is remarkable that even under (H2) the model fails on a non-zero measure set of the initialization space (top-left red region of Fig.

Revised: 2 Left ): in that region, it converges to a classiﬁer assigning a probability of 0 .

Revised: 5 to the true class (dash-dotted curves in Fig.

Revised: 3 Right ).

Original: 3.4 Deeper neural networks
Revised: 3.4 Relaxing Assumption (H2)

Revised: Assumption (H2) states that for any i ∈ I k , x ∈ D k and x (cid:48) / ∈ D k , w i 0 x > 0 and w i 0 x (cid:48) ≤ 0 .

Revised: We now relax it by assuming that a point x 2 in D 2 veriﬁes w 10 x 2 > 0 and we study the evolution of w 1 t .

Revised: We consider updates coming from sampling equally x 1 from D 1 and x 2 from D 2 .

Original: In a similar fashion to the above, we can extend our analysis to a network with N − 1 hidden layers of  the same shape.
Revised: To simplify the analysis, we assume that x T 1 x 2 = 0 and write α t = w 1 t x 1 (equivalent to y t above) and β t = w 1 t x 2 . The triplet ( α t , β t , z t ) satisﬁes the following system of ODEs (see Appendix A.8 for the derivation): [Equation]

Revised: From our relaxation of (H2), we have α 0 , β 0 > 0 .

Revised: Since the system is symmetric under the transformation ( α t , β t , z t ) → ( β t , α t , − z t ) , we can assume z 0 ≥ 0 .

Revised: Due to the ReLU activations

Revised: [Figure]

Revised: of the network, whenever α t or β t reaches zero it becomes constant and its contribution to z t disappears: the model reaches the independent modes of learning regime from Section 3.1 and evolves according to the results above ( e.g. green hyperbolas in the plane β

Revised: = 0 on Fig.

Revised: 3 Left ).

Revised: If β t (resp.

Original: Under some stringent assumptions on the initialization of the network, the classes are learnt independently from one another.
Revised: α t ) reaches 0 at some point, D 1 (resp. D 2 ) becomes the only class activating the neuron. Let us now characterize how the initialization of the network inﬂuences the outcome of learning. Theorem 3.4.

Revised: Letting c :

Revised: = α 20 / (cid:107) x 1 (cid:107) 2 + β 20 / (cid:107) x 2 (cid:107) 2 − z 20 , the solutions of (4) verify for all t ≥ 0 , α 2 t / (cid:107) x 1 (cid:107) 2 + β 2 t / (cid:107) x 2 (cid:107) 2 − z 2 t

Original: We can also show that the ordinary diﬀerential equation veriﬁed by the logit u ( t ) of our network is
Revised: = c . In other terms, they live on hyperboloids (see Fig. 3 Left). If c ≤ 0 , β t reaches 0 at some point during training (Fig.

Revised: 6 of the Appendix).

Revised: If c > 0 , there exists a curve C c (shown in black on Fig.

Revised: 3 Left)

Revised: such that as t → + ∞ , for any initialization ( α 0 , β 0 , z 0 ) ∈ C c : α t → (

Original: [Equation]
Revised: [Equation]

Revised: That curve deﬁnes two regions of the initialization space.

Original:  The full analysis can be found in Appendix B. Solutions of Eq.
Revised: In one, colored yellow on Fig. 3 Left, the trajectories verify α t = 0 for some t . In the other, colored green, β t reaches 0 at some point. Interpretation The proof of the theorem can be found in Appendix A.8.

Original: 4 are plotted in Fig. 
Revised: Fig. 3 Left shows some solutions of (4).

Original: Here too, a sigmoidal shape appears during the learning process. The eﬀect of (cid:107) x (cid:107) reduces as N grows due to the power 2 /N , however, larger values still converge faster ( e.g. the blue and yellow curves).
Revised: Concretely, we see from the equations that the sign of z t determines whether α t and β t increase or decrease, and how fast they do so. z t ’s evolution on the other hand is the result of a competition between α t and β t . If z 0 and/or α 0 are suﬃciently large, β t will decrease fast and long enough to reach 0 at some point (green curves).

Original: Additionally, as noted in Saxe et al. (2013b) for linear networks: the deeper the network, the faster the learning .
Revised: Conversely, for a large β 0 , z t can reach 0 before β t .

Original: This fact is studied in more details in Arora et al. (2018) where depth is shown to accelerate convergence in some cases.
Revised: When that is the case, α t then decreases until it reaches 0 (yellow curves). We plot examples of those behaviors in Fig.

Original: [Figure]
Revised: Right .

Original: T i m e t
Revised: We notice in particular the classic sigmoidal shape appearing, even when Assumption (H2) is violated.

Original: o c o n v e r g e n c e
Revised: This can be explained as follows.

Original: BCEHinge 0 10 20 30 40 50 60 Time 0. C o n f i d e n c e BCE p=0.8 BCE p=0.2 Hinge p=0.8 Hinge p=0.
Revised: In the regime of small initializations (customary in deep learning), the competition between α t , β t and z t happens in a part of parameter space where all the weights are small ( i.e. where the conﬁdence of the network is close to 0 . 5 ). When one class ﬁnally prevails over the other, e.g. β t reaching 0 (orange curve in the plot), the analytical solutions from previous sections apply and the sigmoidal shape arises.

Original:  On the hinge loss
Revised:  On the hinge loss

Original: Recent results in the ﬁeld of generative adversarial networks have resurrected the hinge loss (Miyato et al., 2018).
Revised: Recent results in the ﬁeld of generative adversarial networks have resurrected the hinge loss (Miyato et al., 2018).

Original: While its exact impact on performance is unclear, we run a small experiment to show its ability to generate better samples than the customary cross-entropy (see Fig.
Revised: While its exact impact on performance is unclear, we run a small experiment to show its ability to generate better samples than the customary cross-entropy (see Fig.

Original: 4 and Appendix E).
Revised: 4 and Appendix E).

Original: In order to perhaps uncover reasons behind its eﬃciency, we extend our results to the hinge loss:
Revised: In order to perhaps uncover reasons behind its eﬃciency, we extend our results to the hinge loss:

Original: [Equation]
Revised: [Equation]

Revised: 1.0 Confidence

Revised: T i m e t

Revised: o c o n v e r g e n c e

Revised: BCEHinge 0 10 20 30 40 50 60 Time 0.

Original:  The hinge loss is non diﬀerentiable, but one can simply consider that learning stops as soon as the output of the network reaches 1 (resp. - 1 ) for class D 1 (resp.
Revised: C o n f i d e n c e BCE p=0.8 BCE p=0.2 Hinge p=0.8 Hinge p=0. The hinge loss is non diﬀerentiable, but one can simply consider that learning stops as soon as the output of the network reaches 1 (resp. - 1 ) for class D 1 (resp.

Original: D 2 ).
Revised: D 2 ).

Original: Under the same assumptions than in Theorem 3.2 (some of which can be relaxed, see Appendix C), we have the following result:
Revised: Under the same assumptions than in Theorem 3.2 (some of which can be relaxed, see Appendix C), we have the following result:

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: For x ∈ D 1 (for D 2 it is simply the opposite), the output u ( t ) of the network veriﬁes u ( t ) = min(1 ,
Revised: For x ∈ D 1 (for D 2 it is simply the opposite), the output u ( t ) of the network veriﬁes [Equation] where θ 0 = cosh −

Original: [Equation]
Revised: [Equation]

Original: where θ 0 = cosh − 1 ( y 20 + (cid:107) x (cid:107) 2 z 20 c ) and the left and right equations correspond to c = 0 and c (cid:54) = 0 .
Revised: + [Equation] 2 z 20 c ) and the left and right equations correspond to c = 0 and c (cid:54) = 0 .

Original: Proof.
Revised: Proof.

Original: Simple computations show that the dynamics of the system are governed by y (cid:48)
Revised: Simple computations show that the dynamics of the system are governed by y (cid:48)

Original: t
Revised: t

Original: = x T x z t and z (cid:48)
Revised: = x T x z t and z (cid:48)

Original: t = y t .
Revised: t = y t .

Original: Following the method from Section 3, we see that u (cid:48) ( t )
Revised: Following the method from Section 3, we see that u (cid:48) ( t )

Original: = 2 (cid:107) x (cid:107) u ( t ) in the case
Revised: = 2 (cid:107) x (cid:107) u ( t ) in the case

Original: where c = 0 , leading to to the result.
Revised: where c = 0 , leading to to the result.

Original: When c (cid:54) = 0 , a classic hyperbolic change of variables allows to ﬁnd the solution.
Revised: When c (cid:54) = 0 , a classic hyperbolic change of variables allows to ﬁnd the solution.

Original: Its full derivation can be found in Appendix C.
Revised: Its full derivation can be found in Appendix C.

Original: The learning curves are plotted in Fig.
Revised: The learning curves are plotted in Fig.

Original: 4 Right .
Revised: 4 Right .

Original: We notice a hard sigmoidal shape corresponding to learning stopping when u t reaches 1 .
Revised: We notice a hard sigmoidal shape corresponding to learning stopping when u t reaches 1 .

Original: The comparison to binary cross-entropy (all other parameters kept equal) suggests a much faster convergence.
Revised: The comparison to binary cross-entropy (all other parameters kept equal) suggests a much faster convergence.

Original: With δ the required conﬁdence for our classiﬁer, the time t ∗ required to reach δ can easily be computed.
Revised: With δ the required conﬁdence for our classiﬁer, the time t ∗ required to reach δ can easily be computed.

Original: We plot it for both losses in Fig.
Revised: We plot it for both losses in Fig.

Original: 4 Right which conﬁrms visually that the hinge loss converges much faster.
Revised: 4 Right which conﬁrms visually that the hinge loss converges much faster.

Original: We also notice the expected divergence of t ∗ for the binary cross entropy as δ reaches 1 (training never converges in that case).
Revised: We also notice the expected divergence of t ∗ for the binary cross entropy as δ reaches 1 (training never converges in that case).

Original: We refer the interested reader to Appendix C for a more general treatment of the Hinge loss, which fully relaxes the assumption on the number of points in the classes.
Revised: We refer the interested reader to Appendix C for a more general treatment of the Hinge loss, which fully relaxes the assumption on the number of points in the classes.

Original:  Gradient Starvation
Revised:  Gradient Starvation

Original: In this section, we attempt to quantify the impact of feature frequency inside a given class.
Revised: In this section, we attempt to quantify the impact of feature frequency inside a given class.

Original: We keep our simpliﬁed framework and consider that the input x to our network is composed of two underlying features x 1 ∈ R d 1 and x 2 ∈ R d 2 with d = d 1 + d 2 .
Revised: We keep our simpliﬁed framework and consider that the input x to our network is composed of two underlying features x 1 ∈ R d 1 and x 2 ∈ R d 2 with d = d 1 + d 2 .

Original: We let ( x 1 , x 2 ) ∈
Revised: We let ( x 1 , x 2 ) ∈

Original: R
Revised: R

Original: d denote the concatenation of the vectors x 1 and x 2 .
Revised: d denote the concatenation of the vectors x 1 and x 2 .

Original: We assume that all the points in class D 1 contain the feature x 1 but only a fraction λ of them contains the feature x 2 .
Revised: We assume that all the points in class D 1 contain the feature x 1 but only a fraction λ of them contains the feature x 2 .

Original: This is equivalent to making continuous gradient updates using the vector ( x 1 , x 2 ) with a rate λ and the vector ( x 1 , 0) with a rate 1 − λ .
Revised: This is equivalent to making continuous gradient updates using the vector ( x 1 , x 2 ) with a rate λ and the vector ( x 1 , 0) with a rate 1 − λ .

Original: We also assume that those features are fully informative for
Revised: We also assume that those features are fully informative for

Original: D 1 – i.e. are absent from class D 2 .
Revised: D 1 – i.e. are absent from class D 2 .

Original: A network trained using gradient descent on the dataset we just described has the following property
Revised: A network trained using gradient descent on the dataset we just described has the following property

Original: Even though the feature represented by x 2 is fully informative of the class, the network will not classify a sample containing only x 2 with high conﬁdence.
Revised: Even though the feature represented by x 2 is fully informative of the class, the network will not classify a sample containing only x 2 with high conﬁdence.

Original: It is the result of a phenomenon we coin gradient starvation where the most frequent features starve the gradient for the least frequent ones, resulting in a slower learning of those:
Revised: It is the result of a phenomenon we coin gradient starvation where the most frequent features starve the gradient for the least frequent ones, resulting in a slower learning of those:

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: Let δ be our conﬁdence requirement on class D 1 i.e. training stops as soon as ∀ x ∈ D 1 , P t ( x ∈ D 1 ) ≥ 1 − δ , and let t ∗ denote that instant.
Revised: Let δ be our conﬁdence requirement on class D 1 i.e. training stops as soon as ∀ x ∈ D 1 , P t [Equation] x ∈ D 1 ) ≥ 1 − δ , and let t ∗ denote that instant.

Original: Then, under some mild assumptions,
Revised: Then, under some mild assumptions,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From Lemma 3.1, assumptions (H2-3) are suﬃcient to guarantee independent mode learning as well as positiveness of z t .
Revised: From Lemma 3.1, assumptions (H2-3) are suﬃcient to guarantee independent mode learning as well as positiveness of z t .

Original: We decompose w t =
Revised: We decompose w t =

Original: ( α t x 1 , β t x 2 )+( x ⊥ 1 , x ⊥ 2 ) where x T 1 x ⊥ 1 = x T 2 x ⊥ 2 = 0 , and assume that α 0 ≥ β 0 /λ > 0 (in App.
Revised: ( α t x 1 , β t x 2 )+( x ⊥ 1 , x ⊥ 2 ) where x T 1 x ⊥ 1 = x T 2 x ⊥ 2 = 0 , and assume that α 0 ≥ β 0 /λ > 0 (in App.

Original: D, we relax some of those assumptions and prove an equivalent result).
Revised: D, we relax some of those assumptions and prove an equivalent result).

Original: The evolution equation for w t is w (cid:48) t 
Revised: The evolution equation for w t is w (cid:48) t =

Original: =
Revised: [Equation]

Original: z t x 1+ e ztwtx with x = ( x 1 , x 2 ) (resp.
Revised: 1+ e ztwtx with x = ( x 1 , x 2 ) (resp.

Original: ( x 1 , 0) )
Revised: ( x 1 , 0) )

Original: at an λ (resp.
Revised: at an λ (resp.

Original: 1 − λ ) rate.
Revised: 1 − λ ) rate.

Original: Projecting on x 1 and x 2 gives
Revised: Projecting on x 1 and x 2 gives

Original: [Equation]
Revised: [Equation]

Original: From z t > 0 , we see that β t is an increasing function of time, which guarantees β t > 0 , and
Revised: From z t > 0 , we see that β t is an increasing function of time, which guarantees β t > 0 , and

Original: [Equation]
Revised: [Equation]

Original: This proves that ∀ t ≥ 0 , α t ≥ β t /λ .
Revised: This proves that ∀ t ≥ 0 , α t ≥ β t /λ .

Original: We now consider t ∗ such that z t ∗ α t ∗ = log( 1 − δδ ) .
Revised: We now consider t ∗ such that z t ∗ α t ∗ = log( 1 − δδ ) .

Original: It is the smallest t such that P t (( x 1 , x 2 ) ∈ D 1 ) ≥ P t (( x 1 , 0)
Revised: It is the smallest t such that P t (( x 1 , x 2 ) ∈ D 1 ) ≥ P t (( x 1 , 0)

Original: ∈ D 1 )
Revised: ∈ D 1 )

Original: = 1 − δ , its existence is guaranteed by z t and α t being increasing (we also assume that t = 0 does not verify those (in)equalities).
Revised: = 1 − δ , its existence is guaranteed by z t and α t being increasing (we also assume that t = 0 does not verify those (in)equalities).

Original: We get
Revised: We get

Original: P t ∗ (
Revised: P [Equation]

Original: [Equation] [Equation] 
Revised: [Equation] [Equation] [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Revised: λz t ∗ α t ∗ = 1 + e − λ log( 1 − δδ ) .

Original: As can be seen in Eq.
Revised: As can be seen in Eq.

Original: 7, the presence of α t – which detects feature x 1 – in the denominator of β (cid:48) t greatly reduces its value, thus preventing the network from learning x 2 properly. 
Revised: 7, the presence of α t – which detects feature x 1 – in the denominator of β (cid:48) t greatly reduces its value, thus preventing the network from learning x 2 properly. 1.0 Conf.

Original: [Figure] dataset 
Revised: on the dataset 0.

Original: [Figure]
Revised: C o n f .

Original: [Table]
Revised: o n t

Original: Training Testing Testing (Real)
Revised: h e r a r e f e a t u r e

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: In Fig.
Revised: In Fig.

Original: 5 Left.
Revised: 5 Left.

Original: , we plot for diﬀerent values of λ the conﬁdence of the network when classifying x 2 as a function of its conﬁdence on x 1 (see App.
Revised: , we plot for diﬀerent values of λ the conﬁdence of the network when classifying x 2 as a function of its conﬁdence on x 1 (see App.

Original: D. for more details).
Revised: D. for more details).

Original: The gap between the two is very signiﬁcant: with e.g. λ = 0 .
Revised: The gap between the two is very signiﬁcant: with e.g. λ = 0 .

Original: 1 and δ = 10 − 4 (Table 1), P t ∗ ((0 , x 2 ) ∈ D 1 ) ≤ 72% !
Revised: 1 and δ = 10 − 4 (Table 1), P t ∗ ((0 , x 2 ) ∈ D 1 ) ≤ 72% !

Original: Even though x 2 is exclusively present in
Revised: Even though x 2 is exclusively present in

Original: D 1 , and is thus extremely informative, the network is unable to classify it.
Revised: D 1 , and is thus extremely informative, the network is unable to classify it.

Original: Experiment To validate those ﬁndings empirically, we design an artiﬁcial experiment based on the cats and dogs dataset (Kaggle, 2018).
Revised: Experiment To validate those ﬁndings empirically, we design an artiﬁcial experiment based on the cats and dogs dataset (Kaggle, 2018).

Original: We create a very strong, perfectly discriminative feature by making the dog pictures brighter, and the cat pictures darker.
Revised: We create a very strong, perfectly discriminative feature by making the dog pictures brighter, and the cat pictures darker.

Original: We then train a standard deep neural network to classify the modiﬁed images and measure its performance on the untouched testing set.
Revised: We then train a standard deep neural network to classify the modiﬁed images and measure its performance on the untouched testing set.

Original: Results The results can be seen in Table 2.
Revised: Results The results can be seen in Table 2.

Original: The network perfectly learns to classify both the train and test modiﬁed set, but utterly fails on the real test data.
Revised: The network perfectly learns to classify both the train and test modiﬁed set, but utterly fails on the real test data.

Original: This proves that the handcrafted light feature was learnt by the network, and is used exclusively to classify images.
Revised: This proves that the handcrafted light feature was learnt by the network, and is used exclusively to classify images.

Original: All the features allowing to recognize a cat from a dog are still present in the data, but the low level features ( e.g. the presence of whiskers, how edges combine to form the shape of the animals and so on) are far less frequent than the light intensity, and thus were not learnt.
Revised: All the features allowing to recognize a cat from a dog are still present in the data, but the low level features ( e.g. the presence of whiskers, how edges combine to form the shape of the animals and so on) are far less frequent than the light intensity, and thus were not learnt.

Original: The most frequent feature starved all the others.
Revised: The most frequent feature starved all the others.

Original:  Related work
Revised:  Related work

Original: The learning dynamics of neural networks have been explored for decades.
Revised: The learning dynamics of neural networks have been explored for decades.

Original: Baldi & Hornik (1989) studied the energy landscape of linear networks and the ﬁxed point structure of gradient descent learning in that context.
Revised: Baldi & Hornik (1989) studied the energy landscape of linear networks and the ﬁxed point structure of gradient descent learning in that context.

Original: Heskes & Kappen (1993) developed a theory encompassing stochastic gradient descent and parameter dynamics and wrote down their evolution equations in on-line learning.
Revised: Heskes & Kappen (1993) developed a theory encompassing stochastic gradient descent and parameter dynamics and wrote down their evolution equations in on-line learning.

Original: However, those equations are heavily nonlinear and do not have closed-form solutions in the general case.
Revised: However, those equations are heavily nonlinear and do not have closed-form solutions in the general case.

Original: Saxe et al.
Revised: Saxe et al.

Original: (2013b) study the case of deep linear networks trained with regression.
Revised: (2013b) study the case of deep linear networks trained with regression.

Original: They prove the existence of nonlinear learning phenomena similar to those seen in simulations of nonlinear networks and provide exact solutions to the dynamics of learning in the linear case.
Revised: They prove the existence of nonlinear learning phenomena similar to those seen in simulations of nonlinear networks and provide exact solutions to the dynamics of learning in the linear case.

Original: Some of our results are an extension of theirs to nonlinear networks.
Revised: Some of our results are an extension of theirs to nonlinear networks.

Original: Choromanska et al.
Revised: Choromanska et al.

Original: ; Raghu et al.
Revised: ; Raghu et al.

Original: (2017); Saxe (2015);
Revised: (2017); Saxe (2015);

Original: Yosinski et al.
Revised: Yosinski et al.

Original: (2014) also focus on neural network dynamics, while
Revised: (2014) also focus on neural network dynamics, while

Original: Nacson et al.
Revised: Nacson et al.

Original: Xu et al.
Revised: Xu et al.

Original: ; Soudry et al.
Revised: ; Soudry et al.

Original: (2017) study the convergence rate of learning on separable data.
Revised: (2017) study the convergence rate of learning on separable data.

Original: Arora et al.
Revised: Arora et al.

Original: (2018) prove that overparameterization can lead to faster optimization.
Revised: (2018) prove that overparameterization can lead to faster optimization.

Original: Recent work in the domain of generative adversarial networks (Goodfellow et al., 2014) has shown the resurgence of the hinge loss (Rosasco et al., 2004).
Revised: Recent work in the domain of generative adversarial networks (Goodfellow et al., 2014) has shown the resurgence of the hinge loss (Rosasco et al., 2004).

Original: In particular, part of the success encountered by
Revised: In particular, part of the success encountered by

Original: Miyato et al.
Revised: Miyato et al.

Original: (2018) is due to their use of that speciﬁc loss function.
Revised: (2018) is due to their use of that speciﬁc loss function.

Original: Their main contribution however is a spectral normalization technique that produces state-of-the-art results on image generation.
Revised: Their main contribution however is a spectral normalization technique that produces state-of-the-art results on image generation.

Original: Their paper is part of a larger trend focusing on the spectra of neural network weight matrices and their evolution during learning (Vorontsov et al., 2017; Odena et al., 2018; Pennington et al., 2017).
Revised: Their paper is part of a larger trend focusing on the spectra of neural network weight matrices and their evolution during learning (Vorontsov et al., 2017; Odena et al., 2018; Pennington et al., 2017).

Original: It, nevertheless, remains a poorly understood subject.
Revised: It, nevertheless, remains a poorly understood subject.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2016) performed some experiments proving that deep neural networks expound a socalled implicit regularization .
Revised: (2016) performed some experiments proving that deep neural networks expound a socalled implicit regularization .

Original: Even though they have the ability to entirely memorize the dataset, they still converge to solutions that generalize well.
Revised: Even though they have the ability to entirely memorize the dataset, they still converge to solutions that generalize well.

Original: A variety of explanations for that phenomenon have been advanced: correlation between ﬂatness of minima and generalization (Hochreiter & Schmidhuber, 1997), natural convergence of stochastic gradient descent towards such minima (Kleinberg et al., 2018), built-in hierarchical representations (LeCun et al., 2015), gradient descent naturally protecting against overﬁtting (Advani & Saxe, 2017), and structure of deep networks biasing learning towards simpler functions (Neyshabur et al., 2014; Perez et al., 2018).
Revised: A variety of explanations for that phenomenon have been advanced: correlation between ﬂatness of minima and generalization (Hochreiter & Schmidhuber, 1997), natural convergence of stochastic gradient descent towards such minima (Kleinberg et al., 2018), built-in hierarchical representations (LeCun et al., 2015), gradient descent naturally protecting against overﬁtting (Advani & Saxe, 2017), and structure of deep networks biasing learning towards simpler functions (Neyshabur et al., 2014; Perez et al., 2018).

Original: Our results from Sectionsuggest that gradient descent indeed has a beneﬁcial eﬀect, but can also hurt in some situations.
Revised: Our results from Sectionsuggest that gradient descent indeed has a beneﬁcial eﬀect, but can also hurt in some situations.

Original:  Discussion
Revised:  Discussion

Original: In order to obtain closed form solutions for the learning dynamics, we made the extremely simplifying assumption that each class only contains one point.
Revised: In order to obtain closed form solutions for the learning dynamics, we made the extremely simplifying assumption that each class only contains one point.

Original: We leave overcoming that limitation to future work.
Revised: We leave overcoming that limitation to future work.

Original: In the spirit of the proof in Section 5 where we considered two datapoints, we might be able to obtain upper and lower bounds on the learning dynamics.
Revised: In the spirit of the proof in Section 5 where we considered two datapoints, we might be able to obtain upper and lower bounds on the learning dynamics.

Original: Our comparison between the cross-entropy and the hinge losses reveals fundamental diﬀerences.
Revised: Our comparison between the cross-entropy and the hinge losses reveals fundamental diﬀerences.

Original: It is noteworthy that the hinge loss is an important ingredient of the recently introduced spectral normalization (Miyato et al., 2018).
Revised: It is noteworthy that the hinge loss is an important ingredient of the recently introduced spectral normalization (Miyato et al., 2018).

Original: The fast convergence of networks trained with the hinge loss might in part explain its beneﬁcial impact.
Revised: The fast convergence of networks trained with the hinge loss might in part explain its beneﬁcial impact.

Original: A deeper analysis of the connections between the two would lead to a better understanding of the performance of the algorithm.
Revised: A deeper analysis of the connections between the two would lead to a better understanding of the performance of the algorithm.

Original: In this paper, we introduce the concept of gradient starvation and suggest that it might be a plausible explanation for the generalization abilities of deep neural networks.
Revised: In this paper, we introduce the concept of gradient starvation and suggest that it might be a plausible explanation for the generalization abilities of deep neural networks.

Original: By focusing most of the learning on the frequent features of the dataset, it makes the network ignore the idiosyncrasies of individual datapoints.
Revised: By focusing most of the learning on the frequent features of the dataset, it makes the network ignore the idiosyncrasies of individual datapoints.

Original: That rather desirable property has a downside however: very informative but rare features will not be learnt during training.
Revised: That rather desirable property has a downside however: very informative but rare features will not be learnt during training.

Original: This strongly limits the ability of the network to transfer to diﬀerent data distributions where e.g. the rare feature exists on its own.
Revised: This strongly limits the ability of the network to transfer to diﬀerent data distributions where e.g. the rare feature exists on its own.

Original: Appendix A. A.1 Proof of Lemma 3. 
Revised: Appendix A. A.1 Proof of Lemma 3.

Original: In this section, we prove the following lemma:
Revised: In this section, we prove the following lemma:

Original: Lemma 3.1 For any k ∈ { 1 , 2 } , x ∈ D k and t ≥ 0 , the only non-negative elements of W t x are the ones with an index i ∈
Revised: Lemma 3.1 For any k ∈ { 1 , 2 } , x ∈ D k and t ≥ 0 , the only non-negative elements of W t x are the ones with an index i ∈

Original: I
Revised: I

Original: k .
Revised: k .

Original: The signs of the coordinates of Z t remain the same throughout training.
Revised: The signs of the coordinates of Z t remain the same throughout training.

Original: Proof.
Revised: Proof.

Original: We prove this with a simple induction.
Revised: We prove this with a simple induction.

Original: The claim is true at t = 0 from assumptions (H1-3).
Revised: The claim is true at t = 0 from assumptions (H1-3).

Original: Let us assume that at time set t , the diﬀerent parts of the lemma are true.
Revised: Let us assume that at time set t , the diﬀerent parts of the lemma are true.

Original: The SGD updates to the weights stemming from a single observation x ∈ D k (the extension to a mini-batch is straightforward) with a learning rate α are:
Revised: The SGD updates to the weights stemming from a single observation x ∈ D k (the extension to a mini-batch is straightforward) with a learning rate α are:

Original: [Equation]
Revised: [Equation]

Original: where ⊗ denotes the element-wise product of two vectors and
Revised: where ⊗ denotes the element-wise product of two vectors and

Original: e k ∈ R h is the binary vector with ones on the indices from I k .
Revised: e k ∈ R h is the binary vector with ones on the indices from I k .

Original: δ f ( x ) is the gradient of the loss with respect to the pre-sigmoid output of the network F t ( x ) :
Revised: δ f ( x ) is the gradient of the loss with respect to the pre-sigmoid output of the network F t ( x ) :

Original: δ f ( x ) = 1 { k =1 } − σ ( F t ( x )) .
Revised: δ f ( x ) = 1 { k =1 } − σ ( F t ( x )) .

Original: The updates to Z t are positive on its indices belonging to
Revised: The updates to Z t are positive on its indices belonging to

Original: I 1 , and negative otherwise, proving the second claim of the lemma.
Revised: I 1 , and negative otherwise, proving the second claim of the lemma.

Original: Moving to W t , only its rows and hidden neurons with indices i ∈
Revised: Moving to W t , only its rows and hidden neurons with indices i ∈

Original: I k are modiﬁed.
Revised: I k are modiﬁed.

Original: For an element x (cid:48) ∈ D ,
Revised: For an element x (cid:48) ∈ D ,

Original: W t +1 x (cid:48)
Revised: W t +1 x (cid:48)

Original: =
Revised: =

Original: W t x (cid:48) + αδ f ( x )( (cid:80) i ∈I k z i ) (cid:107) x (cid:107) (cid:48) where z i is the i -th element of Z t .
Revised: W t x (cid:48) + αδ f ( x )( (cid:80) i ∈I k z i ) (cid:107) x (cid:107) (cid:48) where z i is the i -th element of Z t .

Original: By induction, (cid:80) i ∈I k
Revised: By induction, (cid:80) i ∈I k

Original: z i has the same sign as δ f ( x ) (positive on D 1 , negative on D 2 ).
Revised: z i has the same sign as δ f ( x ) (positive on D 1 , negative on D 2 ).

Original: From (H1), (cid:107) x (cid:107) (cid:48) is positive (resp.
Revised: From (H1), (cid:107) x (cid:107) (cid:48) is positive (resp.

Original: negative)
Revised: negative)

Original: if x (cid:48) is in D k (resp.
Revised: if x (cid:48) is in D k (resp.

Original: otherwise).
Revised: otherwise).

Original: The update keeps the k -th neuron active (resp. inactive).
Revised: The update keeps the k -th neuron active (resp. inactive).

Original: A.2 Extension of Lemma 3.1 to multi-class classification
Revised: A.2 Extension of Lemma 3.1 to multi-class classification

Original: Let us consider a simple C -classiﬁcation task.
Revised: Let us consider a simple C -classiﬁcation task.

Original: D =
Revised: D =

Original: { ( x i , y i ) } 1 ≤ i ≤ n ⊂ R d × C is our dataset of vectors/labels where
Revised: { ( x i , y i ) } 1 ≤ i ≤ n ⊂ R d × C is our dataset of vectors/labels where

Original: C denotes both the set of possible labels and its cardinal depending on the context.
Revised: C denotes both the set of possible labels and its cardinal depending on the context.

Original: The classiﬁer we are training is a simple neural network with one hidden layer of C neurons, a ReLU non-linearity and a softmax σ .
Revised: The classiﬁer we are training is a simple neural network with one hidden layer of C neurons, a ReLU non-linearity and a softmax σ .

Original: The full function is written as
Revised: The full function is written as

Original: [Equation]
Revised: [Equation]

Original: where W t (resp.
Revised: where W t (resp.

Original: Z t ) is a C × d (resp.
Revised: Z t ) is a C × d (resp.

Original: C × C ) weight matrix and σ the softmax function.
Revised: C × C ) weight matrix and σ the softmax function.

Original: The subscript + denotes the positive part of a real number.
Revised: The subscript + denotes the positive part of a real number.

Original: P t ( x ) is a C -vector representing the probability that x belongs to each class.
Revised: P t ( x ) is a C -vector representing the probability that x belongs to each class.

Original: The subscript t denotes the state of the element at time step t of training.
Revised: The subscript t denotes the state of the element at time step t of training.

Original: For a given class k , we let D k be the set of vectors belonging to class k .
Revised: For a given class k , we let D k be the set of vectors belonging to class k .

Original: We make the following assumptions, with k (cid:54) =
Revised: We make the following assumptions, with k (cid:54) =

Original: k (cid:48) ∈ C two arbitrary classes
Revised: k (cid:48) ∈ C two arbitrary classes

Original: (H4)
Revised: (H4)

Original: For any x, x (cid:48) ∈ D k , x T x (cid:48) > 0 .
Revised: For any x, x (cid:48) ∈ D k , x T x (cid:48) > 0 .

Original: For any x ∈ D k and x
Revised: For any x ∈ D k and x

Original: (cid:48) ∈ D k (cid:48) , x T x (cid:48) ≤ 0 .
Revised: (cid:48) ∈ D k (cid:48) , x T x (cid:48) ≤ 0 .

Original: (H5)
Revised: (H5)

Original: For any x ∈ D k and x
Revised: For any x ∈ D k and x

Original: (cid:48) ∈
Revised: (cid:48) ∈

Original: D \ D k , w k 0 x > 0 and w k 0 x (cid:48) ≤ 0 , where w k 0 is the k -th row of W 0 .
Revised: D \ D k , w k 0 x > 0 and w k 0 x (cid:48) ≤ 0 , where w k 0 is the k -th row of W 0 .

Original: (H6) Z 0 is initialized randomly to positive numbers on the diagonal, and non-positive elsewhere
Revised: (H6) Z 0 is initialized randomly to positive numbers on the diagonal, and non-positive elsewhere

Original: Those assumptions are straightforward extensions of the ones from the main text.
Revised: Those assumptions are straightforward extensions of the ones from the main text.

Original: We train the classiﬁer using stochastic gradient descent on the cross-entropy loss of our problem.
Revised: We train the classiﬁer using stochastic gradient descent on the cross-entropy loss of our problem.

Original: F t +1 is the state of the neural network after one SGD update to F t .
Revised: F t +1 is the state of the neural network after one SGD update to F t .

Original: Our ﬁrst lemma states that over the course of training, the active neurons of the hidden layer remain the same for each element of the dataset, and the elements of Z t remain of the same sign.
Revised: Our ﬁrst lemma states that over the course of training, the active neurons of the hidden layer remain the same for each element of the dataset, and the elements of Z t remain of the same sign.

Original: Lemma A.1.
Revised: Lemma A.1.

Original: For any k ∈ C , x ∈ D k and t ≥ 0 , the only non-negative element of W t x is its k -th element and all diagonal (resp.
Revised: For any k ∈ C , x ∈ D k and t ≥ 0 , the only non-negative element of W t x is its k -th element and all diagonal (resp.

Original: non-diagonal) elements of Z t are positive (resp. negative).
Revised: non-diagonal) elements of Z t are positive (resp. negative).

Original: Proof.
Revised: Proof.

Original: We prove this with a simple induction.
Revised: We prove this with a simple induction.

Original: The claim is true at t = 0 from assumptions (H4-6).
Revised: The claim is true at t = 0 from assumptions (H4-6).

Original: Let us assume that at time set t , the diﬀerent parts of the lemma are true.
Revised: Let us assume that at time set t , the diﬀerent parts of the lemma are true.

Original: The SGD updates to the weight matrices stemming from a single observation x ∈ D k ∗ (the extension to a mini-batch is straightforward) and a learning rate α are:
Revised: The SGD updates to the weight matrices stemming from a single observation x ∈ D k ∗ (the extension to a mini-batch is straightforward) and a learning rate α are:

Original: [Equation]
Revised: [Equation]

Original: where ⊗ denotes the element-wise product of two vectors and e k ∗ the k ∗ basis vector of R C .
Revised: where ⊗ denotes the element-wise product of two vectors and e k ∗ the k ∗ basis vector of R C .

Original: ∇ y L is the gradient of log F t ( x ) k ∗ (the cross entropy loss for a sample from class k ∗ ) with respect to the output of Z t .
Revised: ∇ y L is the gradient of log F t ( x ) k ∗ (the cross entropy loss for a sample from class k ∗ ) with respect to the output of Z t .

Original: One can show that ( ∇ y L ) k ∗ = 

Revised: One can show that ( ∇ y L ) k ∗ =

Original: − e Ft ( x ) k ∗ (cid:80) j e Ft ( x ) j and ( ∇ y L ) i = − e Ft ( x ) i (cid:80)
Revised: − e Ft ( x ) k ∗ (cid:80) j e Ft ( x ) j and ( ∇ y L ) i = − e Ft ( x ) i (cid:80)

Original: j e Ft ( x ) j for i (cid:54) =
Revised: j e Ft ( x ) j for i (cid:54) =

Original: k ∗ i.e. ∇ y L =
Revised: k ∗ i.e. ∇ y L =

Original: e k ∗ − Y t ( x ) .
Revised: e k ∗ − Y t ( x ) .

Original: The update to Z t is non-negative on the diagonal, and non-positive elsewhere, which proves the second claim of the lemma.
Revised: The update to Z t is non-negative on the diagonal, and non-positive elsewhere, which proves the second claim of the lemma.

Original: As far as W t is concerned, only its k ∗ - th row is modiﬁed.
Revised: As far as W t is concerned, only its k ∗ - th row is modiﬁed.

Original: For an element x (cid:48) ∈ D , W t +1 x (cid:48)
Revised: For an element x (cid:48) ∈ D , W t +1 x (cid:48)

Original: =
Revised: =

Original: W t x (cid:48)
Revised: W t x (cid:48)

Original: + Kx T x (cid:48)
Revised: + Kx T x (cid:48)

Original: e k ∗ where K is the dot product between the k ∗ -th column of Z t and ∇ y L .
Revised: e k ∗ where K is the dot product between the k ∗ -th column of Z t and ∇ y L .

Original: By assumptions on the data, x T x (cid:48) is positive (resp.
Revised: By assumptions on the data, x T x (cid:48) is positive (resp.

Original: negative)
Revised: negative)

Original: if x (cid:48) is in D k ∗ (resp.
Revised: if x (cid:48) is in D k ∗ (resp.

Original: otherwise), so the update keeps the k ∗ -th neuron active (resp. inactive).
Revised: otherwise), so the update keeps the k ∗ -th neuron active (resp. inactive).

Original: The proof can be extended to any number of hidden layers h ≥ C by modifying assumptions (H5-6) using a partition {I 1 , . .
Revised: The proof can be extended to any number of hidden layers h ≥ C by modifying assumptions (H5-6) using a partition {I 1 , . .

Original: , I C } of { 1 , . .
Revised: , I C } of { 1 , . .

Original: , h } similar to the binary classiﬁcation case.
Revised: , h } similar to the binary classiﬁcation case.

Original: Each set I i in the partition describes the neurons active at the initialization of the network for an element x ∈ D i .
Revised: Each set I i in the partition describes the neurons active at the initialization of the network for an element x ∈ D i .

Original: The modiﬁed assumptions are:
Revised: The modiﬁed assumptions are:

Original: (H5’)
Revised: (H5’)

Original: For any i ∈ I
Revised: For any i ∈ I

Original: k , x ∈ D k and x (cid:48) ∈
Revised: k , x ∈ D k and x (cid:48) ∈

Original: D \ D k , w i 0 x > 0 and w i 0 x (cid:48) ≤ 0 . (H6’)
Revised: D \ D k , w i 0 x > 0 and w i 0 x (cid:48) ≤ 0 . (H6’)

Original: For any i ∈ I
Revised: For any i ∈ I

Original: k , j / ∈
Revised: k , j / ∈

Original: I k , ( Z 0 )
Revised: I k , ( Z 0 )

Original: ki > 0 and ( Z 0 ) kj ≤ 0 .
Revised: ki > 0 and ( Z 0 ) kj ≤ 0 .

Original: The lemma then translates to those inequalities remaining true at any time t ≥ 0 .
Revised: The lemma then translates to those inequalities remaining true at any time t ≥ 0 .

Original: A.3 Proofs for Theorem 3. 
Revised: A.3 Proofs for Theorem 3.

Original: In this section we develop the proof of Theorem 3.2 of the main text.
Revised: In this section we develop the proof of Theorem 3.2 of the main text.

Original: Proof.
Revised: Proof.

Original: We consider an update made using x ∈ D 1 .
Revised: We consider an update made using x ∈ D 1 .

Original: Writing y t =
Revised: Writing y t =

Original: w t x , our system follows the system of ordinary diﬀerential equations
Revised: w t x , our system follows the system of ordinary diﬀerential equations

Original: [Equation]
Revised: [Equation]

Original: Writing x T x = (cid:107) x (cid:107) 2 , we see that y t y (cid:48)
Revised: Writing x T x = (cid:107) x (cid:107) 2 , we see that y t y (cid:48)

Original: t
Revised: t

Original: = (cid:107) x (cid:107) 

Revised: = (cid:107) x (cid:107)

Original: z t z (cid:48) t .
Revised: z t z (cid:48) t .

Original: The quantity y 2 t − (cid:107) x (cid:107) 

Revised: The quantity y 2 t − (cid:107) x (cid:107)

Original: z
Revised: z

Original: 2 t is thus an invariant of the problem.
Revised: 2 t is thus an invariant of the problem.

Original: With
Revised: With

Original: [Equation]
Revised: [Equation]

Original: 2 z 20 | , the solutions of Eq.
Revised: 2 z 20 | , the solutions of Eq.

Original: 11 live on hyperbolas of equation
Revised: 11 live on hyperbolas of equation

Original: [Equation]
Revised: [Equation]

Original: We start by treating the case of a degenerate hyperbola c = 0 .
Revised: We start by treating the case of a degenerate hyperbola c = 0 .

Original: We have ∀ t , y t
Revised: We have ∀ t , y t

Original: = (cid:107) x (cid:107) z t
Revised: = (cid:107) x (cid:107) z t

Original: (those quantities are both positive as x ∈ D 1 and (H2-3)).
Revised: (those quantities are both positive as x ∈ D 1 and (H2-3)).

Original: We let u ( t ) :=
Revised: We let u ( t ) :=

Original: z t w t x = z t y t and see by combining the two equations in Eq.
Revised: z t w t x = z t y t and see by combining the two equations in Eq.

Original: 11 that
Revised: 11 that

Original: [Equation]
Revised: [Equation]

Original: For any u f ≥ u 0 , let t =
Revised: For any u f ≥ u 0 , let t =

Original: u < − 1 > ( u f ) ( u is a bijection from R + → [ u 0 , + ∞ [ since its derivative is strictly positive).
Revised: u < − 1 > ( u f ) ( u is a bijection from R + → [ u 0 , + ∞ [ since its derivative is strictly positive).

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where Ei ( x ) = − (cid:82) ∞ − x e − u u du is the exponential integral.
Revised: where Ei ( x ) = − (cid:82) ∞ − x e − u u du is the exponential integral.

Original: In the end, with the superscript < − 1 > denoting the inverse function
Revised: In the end, with the superscript < − 1 > denoting the inverse function

Original: [Equation]
Revised: [Equation]

Original: We let ¯ u denote the function u ( t ) above for (cid:107) x (cid:107)
Revised: We let ¯ u denote the function u ( t ) above for (cid:107) x (cid:107)

Original: = 1 , the solution for (cid:107) x (cid:107) (cid:54)
Revised: = 1 , the solution for (cid:107) x (cid:107) (cid:54)

Original: = 1 can easily be inferred by a rescaling of t in ¯ u .
Revised: = 1 can easily be inferred by a rescaling of t in ¯ u .

Original: For x ∈ D 2 , the system of ODEs is
Revised: For x ∈ D 2 , the system of ODEs is

Original: [Equation]
Revised: [Equation]

Original: the degeneracy assumption becomes y 0 = −(cid:107) x (cid:107) z 

Revised: the degeneracy assumption becomes y 0 = −(cid:107) x (cid:107) z

Original: (we know from (H2)
Revised: (we know from (H2)

Original: that y 0 > 0 and from (H3) that z 0 < 0 ).
Revised: that y 0 > 0 and from (H3) that z 0 < 0 ).

Original: It can be shown similarly that v ( t ) :
Revised: It can be shown similarly that v ( t ) :

Original: =
Revised: =

Original: z t y t veriﬁes the equation v (cid:48) ( t )
Revised: z t y t veriﬁes the equation v (cid:48) ( t )

Original: = 2 (cid:107) x (cid:107) v ( t ) σ ( v ( t )) with a negative initial condition.
Revised: = 2 (cid:107) x (cid:107) v ( t ) σ ( v ( t )) with a negative initial condition.

Original: In other words, u and v follow symmetric trajectories on the positive/negative real line.
Revised: In other words, u and v follow symmetric trajectories on the positive/negative real line.

Original: Below, ¯ u and ¯ v denote those two trajectories for (cid:107) x (cid:107) = 1 and initial conditions u 0 > 0 and v 0 < 0 .
Revised: Below, ¯ u and ¯ v denote those two trajectories for (cid:107) x (cid:107) = 1 and initial conditions u 0 > 0 and v 0 < 0 .

Original: Let us now write p  = | D 1 | / | D | the fraction of points in the dataset belonging to D 1 .
Revised: Let us now write p 1 = | D 1 | / | D | the fraction of points in the dataset belonging to D 1 .

Original: Because we sample randomly from the dataset, this amounts to sampling p  (resp.
Revised: Because we sample randomly from the dataset, this amounts to sampling p 1 (resp.

Original: 1 − p  ) points from D 1 (resp.
Revised: 1 − p 1 ) points from D 1 (resp.

Original: D 2 ) for each time unit during training, ie to rescaling the time axis by p  for D 1 and 1 − p  for D 2 .
Revised: D 2 ) for each time unit during training, ie to rescaling the time axis by p 1 for D 1 and 1 − p 1 for D 2 .

Original: Formally, this allows us to quantify the performance of the network at any time t
Revised: Formally, this allows us to quantify the performance of the network at any time t

Original: [Equation]
Revised: [Equation]

Original: if

Original: [Equation]

Original: which concludes the proof for c = 0 .
Revised: which concludes the proof for c = 0 .

Original: From Eq.
Revised: From Eq.

Original: 14, we also see that
Revised: 14, we also see that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where we used the inequality ∀ y ≥ 0 , e y 2 > y .
Revised: 17) where we used the inequality ∀ y ≥ 0 , e y 2 > y .

Original: It eventually gives us
Revised: It eventually gives us

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  a result in line with convergence rates obtained in Soudry et al.
Revised: 18) a result in line with convergence rates obtained in Soudry et al.

Original: Let us now study the non-degenerate case.
Revised: Let us now study the non-degenerate case.

Original: We apply the classic change of coordinates
Revised: We apply the classic change of coordinates

Original: [Equation]
Revised: [Equation]

Original: Since y 2 t + (cid:107) x (cid:107) 2 z 2 t = c cosh( θ ) and y t
Revised: Since y 2 t + (cid:107) x (cid:107) 2 z 2 t = c cosh( θ ) and y t

Original: z t =c
Revised: z t =

Original: [Equation]
Revised: [Equation]

Original: sinh( θ ) , we see that
Revised: sinh( θ ) , we see that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where the ﬁrst equality used the system of equations Eq. 11.
Revised: where the ﬁrst equality used the system of equations Eq. 11.

Original: This gives us the dynamics of θ as
Revised: This gives us the dynamics of θ as

Original: [Equation]
Revised: [Equation]

Original: with an initial condition θ 0 = cosh − 1 ( y 20 + (cid:107) x (cid:107) 2 z 20 c ) .
Revised: with an initial condition θ 0 = cosh − 1 ( y 20 + (cid:107) x (cid:107) 2 z 20 c ) .

Original: For any θ f ≥ θ 0 , we see that t
Revised: For any θ f ≥ θ 0 , we see that t

Original: = θ < − 1 > ( θ f ) veriﬁes
Revised: = θ < − 1 > ( θ f ) veriﬁes

Original: [Equation]
Revised: [Equation]

Original: There is no closed-form solution for that integral (that we know of).
Revised: There is no closed-form solution for that integral (that we know of).

Original: It can however be computed numerically.
Revised: It can however be computed numerically.

Original: On Fig.
Revised: On Fig.

Original: 2 Right of the main text, we plot the curves for y t and σ ( z t y t ) for diﬀerent values of c and (cid:107) x (cid:107) .
Revised: 2 Right of the main text, we plot the curves for y t and σ ( z t y t ) for diﬀerent values of c and (cid:107) x (cid:107) .

Original: We obtain a sigmoidal shape similar to previously made empirical observations.
Revised: We obtain a sigmoidal shape similar to previously made empirical observations.

Original: We notice in particular that for larger values of (cid:107) x (cid:107) the function converges faster.
Revised: We notice in particular that for larger values of (cid:107) x (cid:107) the function converges faster.

Original: A.4 Extension to h hidden neurons
Revised: A.4 Extension to h hidden neurons

Original: We now extend the result from Theorem 3.2 to the case with h hidden neurons.
Revised: We now extend the result from Theorem 3.2 to the case with h hidden neurons.

Original: Let us still consider an update made on x ∈ D 1 .
Revised: Let us still consider an update made on x ∈ D 1 .

Original: We know from assumptions and by Lemma 3.1 that the only active neurons in the network are indexed by I 1 .
Revised: We know from assumptions and by Lemma 3.1 that the only active neurons in the network are indexed by I 1 .

Original: The network weights follow the evolution equations:
Revised: The network weights follow the evolution equations:

Original: [Equation]
Revised: [Equation]

Original: We similarly deﬁne y it =
Revised: We similarly deﬁne y it =

Original: w it x which brings
Revised: w it x which brings

Original: [Equation]
Revised: [Equation]

Original: The couple ( y it , z it ) follows the same hyperbolic invariance, deﬁned by a constant c i .
Revised: The couple ( y it , z it ) follows the same hyperbolic invariance, deﬁned by a constant c i .

Original: In the case where ∀ i ∈ I 1 , c i = 0 , we see that u it :=
Revised: In the case where ∀ i ∈ I 1 , c i = 0 , we see that u it :=

Original: z it
Revised: z it

Original: y
Revised: y

Original: it veriﬁes
Revised: it veriﬁes

Original: [Equation]
Revised: [Equation]

Original: and a simple summation on i shows that u t := (cid:80) i ∈I 1 u it follows Eq. 13.
Revised: and a simple summation on i shows that u t := (cid:80) i ∈I 1 u it follows Eq. 13.

Original: The dynamics of the logit in this case are identical to the single active neuron case, the only diﬀerence is potentially its initial value.
Revised: The dynamics of the logit in this case are identical to the single active neuron case, the only diﬀerence is potentially its initial value.

Original: A.5 Proof of Corollary 3. 
Revised: A.5 Proof of Corollary 3.

Original: Corollary 3.3 Let δ be the required accuracy on the classiﬁcation task ( i.e. P t ( x ∈ D 1 ) ≥ 1 − δ for x ∈ D 1 ).
Revised: Corollary 3.3 Let δ be the required accuracy on the classiﬁcation task ( i.e. P t ( x ∈ D 1 ) ≥ 1 − δ for x ∈ D 1 ).

Original: Under certain assumptions, the times t ∗ 1 and t ∗ 2 required to reach that accuracy for each classiﬁer verify t ∗ 2 t ∗ 1 ≈ (cid:107) x 1 (cid:107) (cid:107) x 2 (cid:107) p 1 − p where (cid:107) x k (cid:107) is the norm of vector (cid:107) x k (cid:107) from class k .
Revised: Under certain assumptions, the times t ∗ 1 and t ∗ 2 required to reach that accuracy for each classiﬁer verify t ∗ 2 t ∗ 1 ≈ (cid:107) x 1 (cid:107) (cid:107) x 2 (cid:107) p 1 − p where (cid:107) x k (cid:107) is the norm of vector (cid:107) x k (cid:107) from class k .

Original: Proof.
Revised: Proof.

Original: We consider the case c = 0 .
Revised: We consider the case c = 0 .

Original: The classiﬁcation error drops at a rate proportional to (cid:107) x 1 (cid:107) p for D 1 and to (cid:107) x 2 (cid:107) (1 − p ) for D 2 .
Revised: The classiﬁcation error drops at a rate proportional to (cid:107) x 1 (cid:107) p for D 1 and to (cid:107) x 2 (cid:107) (1 − p ) for D 2 .

Original: More precisely, let us assume that u 0 ≤ | v 0 | and look for a classiﬁcation conﬁdence of 1 − δ .
Revised: More precisely, let us assume that u 0 ≤ | v 0 | and look for a classiﬁcation conﬁdence of 1 − δ .

Original: We write u f = σ < − 1 > (1 − δ )
Revised: We write u f = σ < − 1 > (1 − δ )

Original: =
Revised: =

Original: log(1 /δ − 1) , t v = ¯ u < − 1 > ( | v 0 | ) andt ∗ = ¯ u < − 1 > ( u f ) = 1 2(log(log(1 /
Revised: log(1 /δ − 1) , t v = ¯ u < − 1 > ( | v 0 | ) and

Original: [Equation]
Revised: [Equation]

Original: 1) u ) + Ei (log(1 /δ − 1)) − Ei ( u 0 )) , t ∗ represents the time taken to reach conﬁdence δ with an initialization u 0 , and t v the time to reach v 0 starting in u 0 .
Revised: t ∗ represents the time taken to reach conﬁdence δ with an initialization u 0 , and t v the time to reach v 0 starting in u 0 .

Original: We see that
Revised: We see that

Original: [Equation]
Revised: [Equation]

Original: The ratio between the convergence times reads t ∗ 2 t∗ 1 =
Revised: The ratio between the convergence times reads t ∗ 2 t

Original: [Equation]
Revised: [Equation]

Original: p

Original: [Equation]
Revised: [Equation]

Original: 1 − t v t ∗ ) .
Revised: v t ∗ ) .

Original: One sees that if the weight initializations u 0 and v 0 are close ( i.e. t v is small) and the conﬁdence requirement large ( i.e. t ∗ is large), the ratio is approximately (cid:107) x 1 (cid:107) (cid:107) x 2 (cid:107) p 1 − p .A.6 Relaxing assumption (H2) Let us ﬁrst recall that the assumption states: (H2)
Revised: One sees that if the weight initializations u 0 and v 0 are close ( i.e. t v is small) and the conﬁdence requirement large ( i.e. t ∗ is large), the ratio is approximately (cid:107) x 1 (cid:107) (cid:107) x 2 (cid:107) p 1 − p .

Original: For any i ∈ I k , x ∈ D k and x (cid:48) / ∈ D k , w i 0 x > 0 and w i 0 x (cid:48) ≤ 0 .

Original: We now assume that h = 2 and study the evolution of the ﬁrst row of matrix W t , written w t in the following.

Original: We relax assumption (H2) by assuming that there is a point x 2 in D 2 such that w 0 x > 0 .

Original: And we consider updates to w t coming from sampling equally x 1 from D 1 and x 2 .

Original: The evolution equations can be written as

Original: [Equation]

Original: t = x

Original: [Equation]

Original: 1 +

Original: [Equation]

Original: −

Original: [Equation]

Original: In order to make the analysis simpler, we assume that x T 1 x 2 = 0 .

Original: We decompose w t =

Original: α t x 1 + β t x 2 + x ⊥ and get α (cid:48) t =

Original: [-z t 1 + e z t α t , β (cid:48) t = − z t 
-]
Revised: [-z t 1 + e z t α t , β (cid:48) t = − z t 
-]

Original: + e − z t β t , z (cid:48) t

Original: = α t 1 + e z t α t − β t 1 + e − z t β t .

Original: From our relaxation of assumption (H2), we know that β 0 > 0 .

Original: We also have α 0 > 0 and z 0 > 0 .

Original: This means that α t is increasing, and that β t is decreasing until it reaches 0, at which point learning in that direction stops.

Original: A competition appears between α t and β t to determine the sign of z (cid:48) t .

Original: More importantly, α t z t which measures the conﬁdence the network has in x 1 belonging to class 1 evolves according to

Original: [Equation]

Original: The negative term slows down the learning of D 1 until β t reaches 0.

Original: When that happens, we obtain the regime described in the main text.

Original: This observation might be a clue as to why in more complex networks with random initialization, the slow phase of learning can last for a long time: the network needs to build its independent modes of learning.

Original: A.7 Top-left quadrant initialization
Revised: A.6 Top-left quadrant initialization

Original: When the initial conditions of the network verify y 0 = −(cid:107) x [Equation] z 0 , then at all time t , y t = −(cid:107) x [Equation]
Revised: When the initial conditions of the network verify y 0 = −(cid:107) x (cid:107) z 0 , then at all time t , y t = −(cid:107) x (cid:107) z t .

Original: z t .

Original: Plugging that equality in Eq.
Revised: Plugging that equality in Eq.

Original: 11 results in
Revised: 11 results in

Original: [Equation]
Revised: [Equation]

Original: where again u ( t ) =
Revised: where again u ( t ) =

Original: z t y t .
Revised: z t y t .

Original: This means that the logit is negative and increasing.
Revised: This means that the logit is negative and increasing.

Original: Let u 0 < u f < 0 , we see that the time at which u reaches u f veriﬁes
Revised: Let u 0 < u f < 0 , we see that the time at which u reaches u f veriﬁes

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: t diverges to + ∞ as u f tends to 0 from below.

Original: The logit converges to 0 without ever reaching it.

Original: A.8 Relaxing the single datapoint assumption
Revised: A.7 Relaxing the single datapoint assumption

Original: One of the major assumptions made in the main text is the fact that each class contains a single element.
Revised: One of the major assumptions made in the main text is the fact that each class contains a single element.

Original: In this section, we slightly relax it to the case where the points ( { x i } 1 ≤ i ≤ m ⊂ R d ) in a class are all orthogonal to one another 1 (while still verifying Assumption (H1)).
Revised: In this section, we slightly relax it to the case where the points ( { x i } 1 ≤ i ≤ m ⊂ R d ) in a class are all orthogonal to one another 1 (while still verifying Assumption (H1)).

Original: In that case, each presentation of a training vector will only aﬀect w t in the direction of that speciﬁc vector.
Revised: In that case, each presentation of a training vector will only aﬀect w t in the direction of that speciﬁc vector.

Original: Let y it denote w t x
Revised: Let y it denote w t x

Original: i , the unnormalized component of w t along x i .
Revised: i , the unnormalized component of w t along x i .

Original: We consider a batch update on the weights of the neural network.
Revised: We consider a batch update on the weights of the neural network.

Original: In that case:
Revised: In that case:

Original: [Equation]
Revised: [Equation]

Original: Assuming that the vectors all have the same norm (denoted (cid:107) x (cid:107) below) and that the y i 0 are all equal, then that equality remains true at all time (they follow the same update equation).
Revised: Assuming that the vectors all have the same norm (denoted (cid:107) x (cid:107) below) and that the y i 0 are all equal, then that equality remains true at all time (they follow the same update equation).

Original: We let y t denote that value:
Revised: We let y t denote that value:

Original: [Equation]
Revised: [Equation]

Original: A new invariant appears in those equations: c := | my 20 − (cid:107) x
Revised: A new invariant appears in those equations: c := | my 2 0 − [Equation]

Original:  (cid:107) 2{+
+} z 20 | .
Revised: x (cid:107) {+
+} z 2 0 | .

Original: In the case c = [-
-] (the other case can be treated as above), we obtain the following evolution equation for the logit of any point in the class:
Revised: In the case c = [-
-]0 (the other case can be treated as above), we obtain the following evolution equation for the logit of any point in the class:

Original: [Equation]
Revised: [Equation]

Original: We end up with a similar equation than before except for the √ m factor, which boosts the convergence speed.
Revised: We end up with a similar equation than before except for the √ m factor, which boosts the convergence speed.

Original: However, one should not forget that we are now training on a full batch ( i.e. on m points) during each unit of time.
Revised: However, one should not forget that we are now training on a full batch ( i.e. on m points) during each unit of time.

Original: Performing the same number of updates for the single point class would generate a m factor in the convergence speed of u (one √ m factor for each y and z functions).
Revised: Performing the same number of updates for the single point class would generate a m factor in the convergence speed of u (one √ m factor for each y and z functions).

Original: The slower convergence for the more general case can be explained by the fact that each point is making an update on w t in its own direction.
Revised: The slower convergence for the more general case can be explained by the fact that each point is making an update on w t in its own direction.

Original: That direction being orthogonal to all others points makes it useless for their classiﬁcation.
Revised: That direction being orthogonal to all others points makes it useless for their classiﬁcation.

Original: Appendix B.
Revised: A.8 Relaxing assumption (H2)

Revised: Let us ﬁrst recall that the assumption states: [Equation] We now assume that h = 2 and study the evolution of the ﬁrst row w 1 t of matrix W t , written w t in the following.

Revised: We relax assumption (H2) by assuming that there is a point x 2 in D 2 such that w 0 x > 0 .

Revised: And we consider updates to w t coming from sampling equally x 1 from D 1 and x 2 from

Revised: D 2 .

Revised: The evolution equations can be written as

Revised: [Equation]

Revised: In order to make the analysis simpler, we assume that x T 1 x 2 = 0 .

Revised: We write α t =

Revised: w t x 1 and β t = w t x 2 .

Revised: Any component of w 0 orthogonal to both x 1 and x 2 will be untouched by the updates, and does not aﬀect the classiﬁcation performance of the network.

Revised: This gives us

Revised: [Equation]

Revised: β t .

Revised: By assumption, we know that α 0 , β 0 > 0 .

Revised: The system of ODEs (23) is invariant through the transformation ( α t , β t , z t )

Revised: → ( β t , α t , − z t ) so it is suﬃcient to study the case z 0 ≥ 0 .

Revised: Let us now state the theorem from the main text.

Revised: Theorem 3.4.

Revised: Letting c :

Revised: =

Revised: α 20 / (cid:107) x 1 (cid:107) 2 + β 20 / (cid:107) x 2 (cid:107) 2 − z 20 , the solutions of (23) verify for all t ≥ 0 , α 2 t / (cid:107) x 1 (cid:107) 2 + β 2 t / (cid:107) x 2 (cid:107) 2 − z 2 t = c .

Revised: In other terms, they live on hyperboloids (see Fig.

Revised: 3 from the main text and Fig. 6).

Revised: [Figure]

Revised: If c ≤ 0 , β t reaches 0 at some point during training (see Fig. 6).

Revised: If c > 0 , there exists a curve C c on the hyperboloid such that as t → + ∞ , for any initialization ( α 0 , β 0 , z 0 )

Revised: ∈

Revised: C c : α t → (

Revised: [Equation]

Revised: That curve deﬁnes two regions of the initialization space.

Revised: In one (colored yellow on Fig.

Revised: 3 Left) trajectories verify α t = 0 for some

Revised: , in the other (colored green)

Revised: β t reaches 0 at

Revised: Proof.

Revised: It is easy to see from the system of ODEs (23) that ( α 2 t ) (cid:48) / (cid:107) x 1 (cid:107) 2 + ( β 2 t ) (cid:48) / (cid:107) x 2 (cid:107) 2 − ( z 2 t ) (cid:48)

Revised: = 0 , which directly gives the invariance of

Revised: [Equation]

Revised: 2 + β 2 t / (cid:107) x 2 (cid:107) 2 − z 2 t .

Revised: This implies that the trajectories ( α t , β t , z t ) live on hyperboloids.

Revised: If c < 0 , the hyperboloid has two sheets (see Fig. 6).

Revised: In particular, the trajectories verify: z 2 t = α 2 t / (cid:107) x 1 (cid:107) 2 + β 2 t / (cid:107) x 2 (cid:107) 2 − c ≥ − c , which means that z t is bounded away from 0.

Revised: As long as β t ≥ 0 , we have β (cid:48)

Revised: [Equation]

Revised: This implies that β t will reach 0 in a ﬁnite time since it decreases at a rate larger than a strictly positive number.

Revised: At that point, the ReLU ensures that β t does not evolve anymore, and that β t ’s contribution to z t disappears.

Revised: The evolution equations turn into the ones studied in the previous paragraphs, plotted as the green hyperbola in the plane β

Revised: =in Fig.

Revised: If c = 0 , we know that z

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: [Equation]

Revised: 2 ( α t increases as long as z t is positive), so the same argument about β t holds.

Revised: If c > 0 , let us ﬁrst note that the point (( 1 (cid:107)

Revised: [Equation]

Revised: [Equation]

Revised: √

Revised: [Equation]

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: 2 ) − 1 / 2 √ c, 0) belongs to the hyperboloid and is stationary (the three derivatives are 0).

Revised: Classic results on ordinary diﬀerential equations (Tenenbaum & Pollard, 1985) then give the result.

Revised: Finding a closed-form solution to the shape of C c is to the best of our knowledge impossible.

Revised: One can however obtain an approximation by considering a point ((

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: [Equation]

Revised: ) for a small (cid:15) and applying ﬁnite diﬀerence methods to the evolution equations (23) to build the trajectory.

Revised: Appendix B: Deeper Neural Networks

Original: In this section we study the case of a deeper network with N − 1 hidden layers.
Revised: In this section we study the case of a deeper network with N − 1 hidden layers.

Original: Similarly to above, we can prove the existence of independent modes of learning.
Revised: Similarly to above, we can prove the existence of independent modes of learning.

Original: To that end, neurons need to be activated in a disjoint manner from one class to the other.
Revised: To that end, neurons need to be activated in a disjoint manner from one class to the other.

Original: Due to the growing complexity of the interactions between the parameters of the network, this requires very strong assumptions on the initialization of the network and on the shape of the network.
Revised: Due to the growing complexity of the interactions between the parameters of the network, this requires very strong assumptions on the initialization of the network and on the shape of the network.

Original: Assuming that the network is written as
Revised: Assuming that the network is written as

Original: [Equation]
Revised: [Equation]

Original: with W t an h × d matrix and for all 1 ≤ i ≤ N − 2 , Z it an h × h matrix.
Revised: with W t an h × d matrix and for all 1 ≤ i ≤ N − 2 , Z it an h × h matrix.

Original: We maintain assumption (H2) from the main text, and extend (H3) to all Z it by assuming that they are diagonal, and that the j -th element of their diagonal is positive if j ∈ I 1 , negative otherwise.
Revised: We maintain assumption (H2) from the main text, and extend (H3) to all Z it by assuming that they are diagonal, and that the j -th element of their diagonal is positive if j ∈ I 1 , negative otherwise.

Original: To simplify notations, we go back to assuming h = 2 and take an update on x ∈ D 1 .
Revised: To simplify notations, we go back to assuming h = 2 and take an update on x ∈ D 1 .

Original: Only the ﬁrst elements of every matrix are modiﬁed, we write them
Revised: Only the ﬁrst elements of every matrix are modiﬁed, we write them

Original: z it and keep the notations z t , w t and y t .
Revised: z it and keep the notations z t , w t and y t .

Original: We can then write the evolution equations:
Revised: We can then write the evolution equations:

Original: [Equation]
Revised: [Equation]

Original: with u ( t ) =
Revised: with u ( t ) =

Original: z t Π z it y t .
Revised: z t Π z it y t .

Original: Assuming that z 

Revised: Assuming that z

Original: = z 10 = . . .
Revised: = z 10 = . . .

Original: =
Revised: =

Original: z N − [-
-]
Revised: z N − [-
-]1=

Original: [Equation]
Revised: [Equation]

Original: x [Equation] , we see that those equals remain true throughout training.
Revised: x (cid:107) , we see that those equals remain true throughout training.

Original: This gives us
Revised: This gives us

Original: [Equation]
Revised: [Equation]

Original: Combining those equations gives us the ODE veriﬁed by the logit of our system
Revised: Combining those equations gives us the ODE veriﬁed by the logit of our system

Original: [Equation]
Revised: [Equation]

Original: The solution of that equation for N = 4 and N = 8 can be found on Fig.
Revised: The solution of that equation for N = 4 and N = 8 can be found on Fig.

Revised: Here too, a sigmoidal shape appears during the learning process.

Original: 3 of  the main text.
Revised: The eﬀect of (cid:107) x (cid:107) reduces as N grows due to the power 2 /N , however, larger values still converge faster ( e.g. the blue and yellow curves). Additionally, as noted in Saxe et al. (2013b) for linear networks: the deeper the network, the faster the learning . This fact is studied in more details in Arora et al.

Revised: (2018) where depth is shown to accelerate convergence in some cases.

Original: Appendix C. C.1 Proof of Theorem 4. 
Revised: Appendix C: Hinge Loss C.1 Proof of Theorem 4.

Original: In this section we prove Theorem 4.1 from the main text on the dynamics of learning in the case of the Hinge loss:
Revised: In this section we prove Theorem 4.1 from the main text on the dynamics of learning in the case of the Hinge loss:

Original: [Equation]
Revised: [Equation]

Original: Let us consider updates made after observing a point x ∈ D 1 , the converse can be treated similarly with a simple change of sign.
Revised: Let us consider updates made after observing a point x ∈ D 1 , the converse can be treated similarly with a simple change of sign.

Original: The system of ordinary diﬀerential equations veriﬁed by the parameters of our network is:
Revised: The system of ordinary diﬀerential equations veriﬁed by the parameters of our network is:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The same relation between y t and z t appears in those equations than in the cross-entropy case.
Revised: The same relation between y t and z t appears in those equations than in the cross-entropy case.

Original: Deﬁning c similarly, we have u (cid:48) ( t )
Revised: Deﬁning c similarly, we have u (cid:48) ( t )

Original: = 2 (cid:107) x (cid:107) u ( t ) in the case
Revised: = 2 (cid:107) x (cid:107) u ( t ) in the case

Original: where c = 0 , leading to u ( t ) =
Revised: where c = 0 , leading to u ( t ) =

Original: u 0 e 2 (cid:107) x (cid:107) t .
Revised: u 0 e 2 (cid:107) x (cid:107) t .

Original: When c (cid:54) = 0 , the same change of variables can be applied and leads to yt
Revised: When c (cid:54) = 0 , the same change of variables can be applied and leads to y

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: with θ 0 = cosh − 1 ( y 20 + (cid:107) x (cid:107) 2 z 20 c ) .
Revised: with θ 0 = cosh − 1 ( y 20 + (cid:107) x (cid:107) 2 z 20 c ) .

Original: Those equations are only valid until u t reaches 1 2 .
Revised: Those equations are only valid until u t reaches 1 2 .

Original: At that point learning stops, the network has converged.
Revised: At that point learning stops, the network has converged.

Original: If the initialization is such that that condition is already veriﬁed, then the weights will not change as they already solve the task.
Revised: If the initialization is such that that condition is already veriﬁed, then the weights will not change as they already solve the task.

Original: The learning curves along with the initialization diagram can be found in Fig.
Revised: The learning curves along with the initialization diagram can be found in Fig.

Original: We notice a hard sigmoidal shape, corresponding to learning stopping when u t reaches 1 .
Revised: We notice a hard sigmoidal shape, corresponding to learning stopping when u t reaches 1 .

Original: C.2 General treatment
Revised: C.2 General treatment

Original: Let us now consider the general case of a class containing an arbitrary number of points [-C 
-] = { x i } 1 ≤ i ≤ m ⊂ R d .
Revised: Let us now consider the general case of a class containing an arbitrary number of points [-C 
-]D 1 = { x i } 1 ≤ i ≤ m ⊂ R d .

Original: We consider the case of updates done in full batches (standard gradient descent in other words).
Revised: We consider the case of updates done in full batches (standard gradient descent in other words).

Original: In that case, we see that the network obeys the following dynamics:
Revised: In that case, we see that the network obeys the following dynamics:

Original: [Equation]
Revised: [Equation]

Original: Letting X
Revised: Letting X

Original: = m (cid:88) i =
Revised: = m (cid:88) i =

Original: x i denote the sum of all the datapoints in that class, we see that those dynamics boil down to our previous treatment for a single point.
Revised: x i denote the sum of all the datapoints in that class, we see that those dynamics boil down to our previous treatment for a single point.

Original: The same cases appear, depending on the value of c :=
Revised: The same cases appear, depending on the value of c :=

Original: | ( w 0 X ) 2 − (cid:107) X (cid:107) 2 z 20 | .
Revised: | ( w 0 X ) 2 − (cid:107) X (cid:107) 2 z 20 | .

Original: We explicitly treat the c > 0 case.
Revised: We explicitly treat the c > 0 case.

Original: Following the methods above, we see that: w t =
Revised: Following the methods above, we see that: w t =

Original: y t X T (cid:107) X (cid:107) 2 + w ⊥ 0 where y t = √ c cosh( θ 0 2 + (cid:107) X (cid:107) t ) and w T 0 is the component of worthogonal to X (and thus unchanged during training).
Revised: y t X T (cid:107) X (cid:107) 2 + w ⊥ 0 where y t = √ c cosh( θ 0 2 + (cid:107) X (cid:107) t ) and w T 0 is the component of worthogonal to X (and thus unchanged during training).

Original: With z t and u t deﬁned as above (with X instead of x ), an arbitrary example x is then classiﬁed as
Revised: With z t and u t deﬁned as above (with X instead of x ), an arbitrary example x is then classiﬁed as

Original: [Equation]
Revised: [Equation]

Original: Appendix D.
Revised: Appendix D: Gradient Starvation

Original: In this section, we prove a relaxed version of Theorem 5.1 from the main text:
Revised: In this section, we prove a relaxed version of Theorem 5.1 from the main text:

Original: Theorem D.2.
Revised: Theorem D.2.

Original: Let δ be our conﬁdence requirement on class D 1 i.e. the training stops as soon as ∀ x ∈ D 1 , P t ( x ∈ D 1 ) ≥ 1 − δ .
Revised: Let δ be our conﬁdence requirement on class D 1 i.e. the training stops as soon as ∀ x ∈ D 1 , P t ( x ∈ D 1 ) ≥ 1 − δ .

Original: Let t ∗ denote that instant i.e. z t ∗ α t ∗ = log( 1 − δδ ) .
Revised: Let t ∗ denote that instant i.e. z t ∗ α t ∗ = log( 1 − δδ ) .

Original: If β 0 < 0 , the inequality (9) from the main text is valid.
Revised: If β 0 < 0 , the inequality (9) from the main text is valid.

Original: Otherwise, with w 0 =
Revised: Otherwise, with w 0 =

Original: ( α 0 x 1 , β 0 x 2 ) + ( x ⊥ 1 , x ⊥ 2 ) , we have
Revised: ( α 0 x 1 , β 0 x 2 ) + ( x ⊥ 1 , x ⊥ 2 ) , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We start with the β 0 < 0 case.
Revised: We start with the β 0 < 0 case.

Original: If β t ∗ is negative, the result from the main text clearly holds.
Revised: If β t ∗ is negative, the result from the main text clearly holds.

Original: Otherwise, there exists ˜ t < t ∗ such that β ˜ t = 

Revised: Otherwise, there exists ˜ t < t ∗ such that β ˜ t =

Original: ( β t is increasing).
Revised: ( β t is increasing).

Original: The proof of Theorem 5.1 from the main text can then directly be applied to [˜ t, t ∗ ] .
Revised: The proof of Theorem 5.1 from the main text can then directly be applied to [˜ t, t ∗ ] .

Original: If β 0 > 0 , the inequality on α (cid:48) t and β (cid:48) t holds and gives β t ≤ β 0 + ( α t − α 0 ) λ .
Revised: If β 0 > 0 , the inequality on α (cid:48) t and β (cid:48) t holds and gives β t ≤ β 0 + ( α t − α 0 ) λ .

Original: Plugging it into P t ∗ ((0 , x 2 ) ∈ D 1 ) concludes our proof.
Revised: Plugging it into P t ∗ ((0 , x 2 ) ∈ D 1 ) concludes our proof.

Original: In the main text, we assume that β 0 − αα 0 < 0 and obtain a bound on the conﬁdence which is independent from α 0 and β 0 .
Revised: In the main text, we assume that β 0 − αα 0 < 0 and obtain a bound on the conﬁdence which is independent from α 0 and β 0 .

Original: Using that bound allows to obtain Fig. 7, but is partly unfair as the initialization of the network is already favoring the strong feature.
Revised: Using that bound allows to obtain Fig. 9, but is partly unfair as the initialization of the network is already favoring the strong feature.

Original: However, we note that under small random initialization z t ∗ and α t ∗ are of the same order of magnitude and ( β 0 − αα 0 ) is very small compared to log( 1 − δδ ) .
Revised: However, we note that under small random initialization z t ∗ and α t ∗ are of the same order of magnitude and ( β 0 − αα 0 ) is very small compared to log( 1 − δδ ) .

Original: The additional term in the denominator thus has a limited eﬀect on the exponential, gradient starvation is still happening (a fact conﬁrmed by the experiment on the cats and dogs dataset).
Revised: The additional term in the denominator thus has a limited eﬀect on the exponential, gradient starvation is still happening (a fact conﬁrmed by the experiment on the cats and dogs dataset).

Original: In the main text, Fig.
Revised: In the main text, Fig.

Original: 5 plots the upper bound for a fair initialization α 

Revised: 5 plots the upper bound for a fair initialization α

Original: =
Revised: =

Original: β 0 = 0 .
Revised: β 0 = 0 .

Original: 1 (in that case, we need to assume that z t ∗ =
Revised: 1 (in that case, we need to assume that z t ∗ =

Original: α t ∗ ).
Revised: α t ∗ ).

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Appendix E. E.1 Mixture of Gaussian experiment
Revised: Appendix E: Experimental Details E.1 Mixture of Gaussian experiment

Original: In this experiment, the data is constructed using eight independent Gaussian distributions around a unit circle.
Revised: In this experiment, the data is constructed using eight independent Gaussian distributions around a unit circle.

Original: The variance of each Gaussian is chosen such that all eight modes of the data are separated by regions of low data probability, but still contain a reasonable amount of variance.
Revised: The variance of each Gaussian is chosen such that all eight modes of the data are separated by regions of low data probability, but still contain a reasonable amount of variance.

Original: This simple experiment resembles multi-modal datasets.
Revised: This simple experiment resembles multi-modal datasets.

Original: Although this task might seem simple, in practice many generative adversarial networks fail to capture all the modes.
Revised: Although this task might seem simple, in practice many generative adversarial networks fail to capture all the modes.

Original: This problem is generally known as mode collapse .
Revised: This problem is generally known as mode collapse .

Original: As shown in the main text, using the hinge loss instead of the common binary cross-entropy loss alleviates the problem signiﬁcantly.
Revised: As shown in the main text, using the hinge loss instead of the common binary cross-entropy loss alleviates the problem signiﬁcantly.

Original: The architectures used for the generator and discriminator both consist of four hidden layers where each layer has 256 hidden units.
Revised: The architectures used for the generator and discriminator both consist of four hidden layers where each layer has 256 hidden units.

Original: As a common choice, a ReLU is used as the non-linearity function for hidden units.
Revised: As a common choice, a ReLU is used as the non-linearity function for hidden units.

Original: The length of the noise input vector is 128.
Revised: The length of the noise input vector is 128.

Original: The
Revised: The

Original: Adam optimizer (Kingma & Ba, 2014) was applied during training with α = 10 − 4 , β 1 = 0 .
Revised: Adam optimizer (Kingma & Ba, 2014) was applied during training with α = 10 − 4 , β 1 = 0 .

Original: 5 and β 2 = 0 .
Revised: 5 and β 2 = 0 .

Original: The PyTorch framework (Paszke et al., 2017) was used to conduct the experiment.
Revised: The PyTorch framework (Paszke et al., 2017) was used to conduct the experiment.

Original: E.2 Dogs vs. Cats classification with light effect
Revised: E.2 Dogs vs. Cats classification with light effect

Original: For the purpose of highlighting the fact that the most frequent feature starved all the others , we conducted an experiment on a classiﬁcation task.
Revised: For the purpose of highlighting the fact that the most frequent feature starved all the others , we conducted an experiment on a classiﬁcation task.

Original: We modiﬁed the cats and dogs dataset (Kaggle, 2018) by setting the cats images to be lighter than the dogs images.
Revised: We modiﬁed the cats and dogs dataset (Kaggle, 2018) by setting the cats images to be lighter than the dogs images.

Original: To do so, each pixel in a cat image is scaled to be between 0 and 127 while each pixel in a dog image is scaled to be between 128 and 255.
Revised: To do so, each pixel in a cat image is scaled to be between 0 and 127 while each pixel in a dog image is scaled to be between 128 and 255.

Original: The dataset consists of 12500 images of each class.
Revised: The dataset consists of 12500 images of each class.

Original: The classiﬁer has an architecture similar to VGG16 (Simonyan & Zisserman, 2014).
Revised: The classiﬁer has an architecture similar to VGG16 (Simonyan & Zisserman, 2014).

Original: In order to isolate the eﬀect of the induced bias, no regularization was applied.
Revised: In order to isolate the eﬀect of the induced bias, no regularization was applied.

Original: The Adam optimizer was applied here as well during training with α = 10 − 4 , β 1 = 0 .
Revised: The Adam optimizer was applied here as well during training with α = 10 − 4 , β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: The PyTorch framework was used to conduct the experiment.
Revised: The PyTorch framework was used to conduct the experiment.