Original: T HE I NDUCTIVE B IAS OF R E LU N ETWORKS ON O RTHOGONALLY S EPARABLE D ATA
Revised: T HE I NDUCTIVE B IAS OF R E LU N ETWORKS ON O RTHOGONALLY S EPARABLE D ATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: We study the inductive bias of two-layer ReLU networks trained by gradient ﬂow.
Revised: We study the inductive bias of two-layer ReLU networks trained by gradient ﬂow.

Original: We identify a class of easy-to-learn (‘orthogonally separable’) datasets, and characterise the solution that ReLU networks trained on such datasets converge to.
Revised: We identify a class of easy-to-learn (‘orthogonally separable’) datasets, and characterise the solution that ReLU networks trained on such datasets converge to.

Original: Irrespective of network width, the solution turns out to be a combination of two max-margin classiﬁers: one corresponding to the positive data subset and one corresponding to the negative data subset.
Revised: Irrespective of network width, the solution turns out to be a combination of two max-margin classiﬁers: one corresponding to the positive data subset and one corresponding to the negative data subset.

Original: The proof is based on the little-known concept of extremal sectors, for which we prove a number of properties in the context of orthogonal separability.
Revised: The proof is based on the little-known concept of extremal sectors, for which we prove a number of properties in the context of orthogonal separability.

Original: In particular, we prove stationarity of activation patterns from some time T onwards, which enables a reduction of the ReLU network to an ensemble of linear subnetworks.
Revised: In particular, we prove stationarity of activation patterns from some time T onwards, which enables a reduction of the ReLU network to an ensemble of linear subnetworks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: This paper is motivated by the problem of understanding the inductive bias of ReLU networks, or to put it plainly, understanding what it is that neural networks learn.
Revised: This paper is motivated by the problem of understanding the inductive bias of ReLU networks, or to put it plainly, understanding what it is that neural networks learn.

Original: This is a fundamental open question in neural network theory; it is also a crucial part of understanding how neural networks behave on previously unseen data (generalisation) and it could ultimately lead to rigorous a priori guarantees on neural nets’ behaviour.
Revised: This is a fundamental open question in neural network theory; it is also a crucial part of understanding how neural networks behave on previously unseen data (generalisation) and it could ultimately lead to rigorous a priori guarantees on neural nets’ behaviour.

Original: For a long time, the dominant way of thinking about machine learning systems was as minimisers of the empirical risk (Vapnik, 1998; Shalev-Shwartz & Ben-David, 2014).
Revised: For a long time, the dominant way of thinking about machine learning systems was as minimisers of the empirical risk (Vapnik, 1998; Shalev-Shwartz & Ben-David, 2014).

Original: However, this paradigm has turned out to be insufﬁcient for understanding deep learning, where many empirical risk minimisers exist, often with vastly different generalisation properties.
Revised: However, this paradigm has turned out to be insufﬁcient for understanding deep learning, where many empirical risk minimisers exist, often with vastly different generalisation properties.

Original: To understand deep networks, we therefore need a more ﬁne-grained notion of ‘what the model learns’.
Revised: To understand deep networks, we therefore need a more ﬁne-grained notion of ‘what the model learns’.

Original: This has motivated the study of the implicit bias of the training procedure – the ways in which the training algorithm inﬂuences which of the empirical risk minimisers is attained.
Revised: This has motivated the study of the implicit bias of the training procedure – the ways in which the training algorithm inﬂuences which of the empirical risk minimisers is attained.

Original: This is a productive research area, and the implicit bias has already been worked out for many linear models.
Revised: This is a productive research area, and the implicit bias has already been worked out for many linear models.

Original: Notably,
Revised: Notably,

Original: Soudry et al.
Revised: Soudry et al.

Original: (2018) consider a logistic regression classiﬁer trained on linearly separable data, and show that the normalised weight vector converges to the max-margin direction.
Revised: (2018) consider a logistic regression classiﬁer trained on linearly separable data, and show that the normalised weight vector converges to the max-margin direction.

Original: Building on their work, Ji & Telgarsky (2019a) consider deep linear networks, also trained on linearly separable data, and show that the normalised end-to-end weight vector converges to the max-margin direction.
Revised: Building on their work, Ji & Telgarsky (2019a) consider deep linear networks, also trained on linearly separable data, and show that the normalised end-to-end weight vector converges to the max-margin direction.

Original: They in fact show that all ﬁrst-layer neurons converge to the same ‘canonical neuron’ (which points in the max-margin direction).
Revised: They in fact show that all ﬁrst-layer neurons converge to the same ‘canonical neuron’ (which points in the max-margin direction).

Original: Although such impressive progress on linear models has spurred attempts at nonlinear extensions, the problem is much harder and analogous nonlinear results have been elusive.
Revised: Although such impressive progress on linear models has spurred attempts at nonlinear extensions, the problem is much harder and analogous nonlinear results have been elusive.

Original: In this work, we provide the ﬁrst such inductive-bias result for ReLU networks trained on ‘easy’ datasets.
Revised: In this work, we provide the ﬁrst such inductive-bias result for ReLU networks trained on ‘easy’ datasets.

Original: Speciﬁcally, we
Revised: Speciﬁcally, we

Original: • propose orthogonal separability of datasets as a stronger form of linear separability that facilitates the study of ReLU network training, • prove that a two-layer ReLU network trained on an orthogonally separable dataset learns a function with two distinct groups of neurons, where all neurons in each group converge to the same ‘canonical neuron’, • characterise the directions of the canonical neurons, which turn out to be the max-margin directions for the positive and the negative data subset.
Revised: • propose orthogonal separability of datasets as a stronger form of linear separability that facilitates the study of ReLU network training, • prove that a two-layer ReLU network trained on an orthogonally separable dataset learns a function with two distinct groups of neurons, where all neurons in each group converge to the same ‘canonical neuron’, • characterise the directions of the canonical neurons, which turn out to be the max-margin directions for the positive and the negative data subset.

Original: The proof is based on the recently introduced concept of extremal sectors (Maennel et al., 2018) which govern the early phase of training.
Revised: The proof is based on the recently introduced concept of extremal sectors (Maennel et al., 2018) which govern the early phase of training.

Original: Our main technical contributions are a precise characterisation of extremal sectors for orthogonally separable datasets, and an invariance property which ensures that the network’s activation pattern becomes ﬁxed at some point during training.
Revised: Our main technical contributions are a precise characterisation of extremal sectors for orthogonally separable datasets, and an invariance property which ensures that the network’s activation pattern becomes ﬁxed at some point during training.

Original: The latter allows us to treat ReLU networks late in training as ensembles of linear networks, which are much better understood.
Revised: The latter allows us to treat ReLU networks late in training as ensembles of linear networks, which are much better understood.

Original: We hope that a similar proof strategy could be useful in other contexts as well.
Revised: We hope that a similar proof strategy could be useful in other contexts as well.

Original:  F ORMAL SETTING
Revised:  S ETTING AND ASSUMPTIONS

Original: In this section, we introduce the learning scenario including the assumptions we make about the dataset, the model, and the training procedure.
Revised: In this section, we introduce the learning scenario including the assumptions we make about the dataset, the model, and the training procedure.

Original: We consider binary classiﬁcation.
Revised: We consider binary classiﬁcation.

Original: Denote the training data { ( x i , y i ) } ni =1 with x i ∈ R d and y i ∈ {± 1 } for all i ∈ [ n ] .
Revised: Denote the training data { ( x i , y i ) } ni =1 with x i ∈ R d and y i ∈ {± 1 } for all i ∈ [ n ] .

Original: We denote by X ∈ R d × n the matrix with { x i } as columns and by y ∈ R n the vector with { y i } as entries.
Revised: We denote by X ∈ R d × n the matrix with { x i } as columns and by y ∈ R n the vector with { y i } as entries.

Original: Orthogonally separable data.
Revised: Orthogonally separable data.

Original: A binary classiﬁcation dataset ( X , y ) is called orthogonally separable if for all i, j ∈ [ n ] ,
Revised: A binary classiﬁcation dataset ( X , y ) is called orthogonally separable if for all i, j ∈ [ n ] ,

Original: [Equation]
Revised: [Equation]

Original: In other words, a dataset is orthogonally separable iff it is linearly separable, and any training example x i can serve as a linear separator.
Revised: In other words, a dataset is orthogonally separable iff it is linearly separable, and any training example x i can serve as a linear separator.

Original: Geometrically, this means that examples with y i =
Revised: Geometrically, this means that examples with y i =

Original: 1 (‘positive examples’) and examples with y i = − 1 (‘negative examples’) lie in opposite orthants.
Revised: 1 (‘positive examples’) and examples with y i = − 1 (‘negative examples’) lie in opposite orthants.

Original: Two-layer ReLU networks.
Revised: Two-layer ReLU networks.

Original: We deﬁne two-layer width-
Revised: We deﬁne two-layer width-

Original: p fully-connected ReLU networks, parameterised by θ (cid:44) { W , a } , as
Revised: p fully-connected ReLU networks, parameterised by θ (cid:44) { W , a } , as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where W (cid:44) [ w 1 , . . .
Revised: where W (cid:44) [ w 1 , . . .

Original: w p ] (cid:124) ∈
Revised: w p ] (cid:124) ∈

Original: R p ×
Revised: R p ×

Original: d and a (cid:44) [ a 1 , . .
Revised: d and a (cid:44) [ a 1 , . .

Original: . , a p ]
Revised: . , a p ]

Original: (cid:124) ∈
Revised: (cid:124) ∈

Original: R p are the ﬁrst- and second-layer weights of the network, and ρ is the element-wise ReLU function, ρ ( z )
Revised: R p are the ﬁrst- and second-layer weights of the network, and ρ is the element-wise ReLU function, ρ ( z )

Original: i = max { 0 , z i } .
Revised: i = max { 0 , z i } .

Original: We will often view the network as a collection of neurons, { ( a j , w j ) } pj =1 .
Revised: We will often view the network as a collection of neurons, { ( a j , w j ) } pj =1 .

Original: Cross-entropy loss.
Revised: Cross-entropy loss.

Original: We assume a training loss of the form
Revised: We assume a training loss of the form

Original: [Equation]
Revised: [Equation]

Original: this is the standard empirical cross-entropy loss.
Revised: this is the standard empirical cross-entropy loss.

Original: More generally, our results hold when the loss is differentiable, (cid:96) (cid:48)
Revised: More generally, our results hold when the loss is differentiable, (cid:96) (cid:48)

Original: i is bounded and Lipschitz continuous, and satisﬁes − y i (cid:96) (cid:48)
Revised: i is bounded and Lipschitz continuous, and satisﬁes − y i (cid:96) (cid:48)

Original: i ( u ) > 0 for all u ∈ R .
Revised: i ( u ) > 0 for all u ∈ R .

Original: Gradient ﬂow training.
Revised: Gradient ﬂow training.

Original: We assume the loss is optimised by gradient descent with inﬁnitesimally small step size, also known as gradient ﬂow.
Revised: We assume the loss is optimised by gradient descent with inﬁnitesimally small step size, also known as gradient ﬂow.

Original: Under the gradient ﬂow dynamics, the parameter trajectory is an absolutely continuous curve { θ ( t ) | t ≥ 0 } satisfying the differential inclusion
Revised: Under the gradient ﬂow dynamics, the parameter trajectory is an absolutely continuous curve { θ ( t ) | t ≥ 0 } satisfying the differential inclusion

Original: [Equation]
Revised: [Equation]

Original: almost all
Revised: almost all

Original: [Equation]
Revised: [Equation]

Original: where ∂(cid:96) denotes the Clarke subdifferential (Clarke, 1975; Clarke et al., 2008) of (cid:96) , an extension of the gradient to not-everywhere differentiable functions,
Revised: where ∂(cid:96) denotes the Clarke subdifferential (Clarke, 1975; Clarke et al., 2008) of (cid:96) , an extension of the gradient to not-everywhere differentiable functions,

Original: [Equation]
Revised: [Equation]

Original: θ ( t ) is the value of the parameters at time t , and we will use the sufﬁx ( t ) more generally to denote the value of some function of θ at time t .
Revised: θ ( t ) is the value of the parameters at time t , and we will use the sufﬁx ( t ) more generally to denote the value of some function of θ at time t .

Original: Near-zero balanced initialisation.
Revised: Near-zero balanced initialisation.

Original: We assume that the neurons { w j } are initialised iid from the
Revised: We assume that the neurons { w j } are initialised iid from the

Original: Gaussian distribution and then rescaled such that (cid:107) w j (cid:107) ≤ λ , where λ > 0 is a small constant.
Revised: Gaussian distribution and then rescaled such that (cid:107) w j (cid:107) ≤ λ , where λ > 0 is a small constant.

Original: That is, w j = λ j v j / (cid:107) v j (cid:107) forsome λ j ≤ λ and v j iid ∼ N ( 0 , I )  .
Revised: That is, w j = λ j v j / (cid:107) v j (cid:107) for v j iid ∼ N ( 0 , I ) and arbitrary λ j satisfying λ j ∈ (0 , λ ] .

Original: We also assume that a j ∈ {± λ j } .
Revised: We also assume that a j ∈ {± λ j } .

Original: These technical conditions ensure that the neurons are balanced and small in size, (cid:107) w j (cid:107) =
Revised: These technical conditions ensure that the neurons are balanced and small in size, (cid:107) w j (cid:107) =

Original: | a j | ≈ 0 , which simpliﬁes the calculations involved in the analysis of gradient ﬂow.
Revised: | a j | ≈ 0 , which simpliﬁes the calculations involved in the analysis of gradient ﬂow.

Revised: Support examples span the full space.

Revised: We assume that the support examples of the positive data subset { x i | y i = 1 } span the entire R d , and similarly that the support examples of the negative data subset { x i | y

Revised: i = − 1 } span R d .

Revised: (We formally deﬁne support examples after introducing some more notation below.)

Original:  M AIN RESULT
Revised:  M AIN RESULT

Original: Under the assumptions of Section 2 (and one additional assumption), the network converges to a linear combination of two max-margin neurons.
Revised: Under the assumptions of Section 2, the network converges to a linear combination of two maxmargin neurons.

Original: Speciﬁcally, given a dataset ( X , y ) , deﬁne the positive and the negative max-margin vectors w + , w − ∈ R d as
Revised: Speciﬁcally, given a dataset ( X , y ) , deﬁne the positive and the negative maxmargin vectors w + , w − ∈ R d as

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: We call examples which attain equality in eqs.
Revised: We call examples which attain equality in eqs.

Original: (6) and (7) positive support examples and negative support examples respectively.
Revised: (6) and (7) positive support examples and negative support examples respectively.

Original: We now state the main result.
Revised: We now state the main result.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let f θ be a two-layer
Revised: Let f θ be a two-layer

Original: width- p ReLU network trained by gradient ﬂow with the crossentropy loss, initialised near-zero and balanced.
Revised: width- p ReLU network trained by gradient ﬂow with the crossentropy loss, initialised near-zero and balanced.

Original: Consider an orthogonally separable dataset ( X , y ) such that its positive support examples span R d , and its negative support examples also span R d .
Revised: Consider an orthogonally separable dataset ( X , y ) such that its positive support examples span R d , and its negative support examples also span R d .

Original: For almost all such datasets  and with probability 1 − 1 / 2 p over the random initialisation,
Revised: For almost all such datasets 2 and with probability 1 − 1 / 2 p over the random initialisation,

Original: [Equation]
Revised: [Equation]

Original: for some u , z ∈ R p + such that either u i = 

Revised: for some u , z ∈ R p + such that either u i =

Original: or z i = 0 for all i ∈
Revised: or z i = 0 for all i ∈

Original: [ p ] .
Revised: [ p ] .

Revised: Also,

Revised: [Equation]

Original:  The theorem says that each neuron (row of W ), properly normalised, converges either to a scalar multiple of the positive max-margin direction, u i w + , or to a scalar multiple of the negative maxmargin direction, z i w − .
Revised: as t → ∞ [Equation] [Equation] The theorem says that each neuron (row of W ), properly normalised, converges either to a scalar multiple of the positive max-margin direction, u i w + , or to a scalar multiple of the negative maxmargin direction, z i w − .

Original: In other words, there are asymptotically only two distinct types of neurons, and the network could in principle be pruned down to a width of just two.
Revised: In other words, there are asymptotically only two distinct types of neurons, and the network could in principle be pruned down to a width of just two.

Original: These two ‘canonical neurons’ moreover have an explicit characterisation, given by eqs.
Revised: These two ‘canonical neurons’ moreover have an explicit characterisation, given by eqs.

Original: (6) and (7).
Revised: (6) and (7).

Revised: As for the second-layer weights, the magnitude of each a j equals the norm of the respective w j , and the sign of a j is +1 if w j approaches w + and − 1 if w j approaches w − .

Original: The following corollary expresses the above in terms of the function learnt by the network.
Revised: The following corollary summarises the above in terms of the function learnt by the network.

Original: Corollary 1.
Revised: Corollary 1.

Original: Under the conditions of Theorem 1, there exist constants u, z ≥ 0 such that
Revised: Under the conditions of Theorem 1, there exist constants u, z ≥ 0 such that

Original: [Equation]
Revised: [Equation]

Original: 3. D ISCUSSION OF ASSUMPTIONS
Revised: 3. D ISCUSSION OF ASSUMPTIONS

Original: Many of our assumptions are technical, serving to simplify the analysis while detracting little from the result’s relevance 2 .
Revised: Many of our assumptions are technical, serving to simplify the analysis while detracting little from the result’s relevance 3 .

Original: These include inﬁnitesimal step size (gradient ﬂow), balancedness at initialisation and the condition on support span.
Revised: These include inﬁnitesimal step size (gradient ﬂow), balancedness at initialisation and the condition on support span.

Original: The ﬁrst two could potentially be relaxed to their approximate counterparts, i.e. gradient descent with a small constant step size and approximate balancedness (Arora et al., 2019).
Revised: The ﬁrst two could potentially be relaxed to their approximate counterparts, i.e. gradient descent with a small constant step size and approximate balancedness (Arora et al., 2019).

Original: The assumption that support vectors span R d comes from Ji &
Revised: The assumption that support vectors span R d comes from Ji &

Original: Telgarsky (2019a); Soudry et al.
Revised: Telgarsky (2019a); Soudry et al.

Original: It seems to us that it could be lifted, though we have not investigated this possibility in depth.
Revised: It seems to us that it could be lifted, though we have not investigated this possibility in depth.

Original: Two assumptions that deserve more attention are near-zero initialisation and orthogonal separability; both are crucial for the result to hold.
Revised: Two assumptions that deserve more attention are near-zero initialisation and orthogonal separability; both are crucial for the result to hold.

Original: Near-zero initialisation grants neurons high directional  mobility early in training, allowing them to cluster close to the canonical directions.
Revised: Near-zero initialisation grants neurons high directional We verify experimentally in Section 5.1 that these assumptions are indeed not crucial. mobility early in training, allowing them to cluster close to the canonical directions.

Original: Orthogonal separability ensures that the canonical directions are ‘easy to ﬁnd’ by local descent.
Revised: Orthogonal separability ensures that the canonical directions are ‘easy to ﬁnd’ by local descent.

Original: In prior work, which considered linear networks, this role is fulﬁlled by linear separability.
Revised: In prior work, which considered linear networks, this role is fulﬁlled by linear separability.

Original: The reason we need a stronger condition is that ReLU updates are more local compared to linear updates: a linear neuron takes into account all examples in the training set, whereas a ReLU neuron updates only on examples in its positive half-plane (its active examples).
Revised: The reason we need a stronger condition is that ReLU updates are more local compared to linear updates: a linear neuron takes into account all examples in the training set, whereas a ReLU neuron updates only on examples in its positive half-plane (its active examples).

Original: ReLU neurons therefore easily get stuck in a variety of directions, unless the data is highly structured.
Revised: ReLU neurons therefore easily get stuck in a variety of directions, unless the data is highly structured.

Original:  P ROOF SKETCH
Revised:  P ROOF SKETCH

Original: In the analysis, we distinguish between two phases of training.
Revised: In the analysis, we distinguish between two phases of training.

Original: The ﬁrst phase takes place close to the origin, (cid:107) θ (cid:107) ≈ 0 .
Revised: The ﬁrst phase takes place close to the origin, (cid:107) θ (cid:107) ≈ 0 .

Original: In this phase, while neurons move little in the absolute sense, they converge in direction to certain regions of the weight space called extremal sectors .
Revised: In this phase, while neurons move little in the absolute sense, they converge in direction to certain regions of the weight space called extremal sectors .

Original: 4. C ONVERGENCE TO EXTREMAL SECTORS
Revised: 4. C ONVERGENCE TO EXTREMAL SECTORS

Original: (All deﬁnitions and results in this subsection are by Maennel et al.
Revised: (All deﬁnitions and results in this subsection are by Maennel et al.

Original: We need them later on.)
Revised: We need them later on.)

Original: Sectors are regions in weight space corresponding to different activation patterns.
Revised: Sectors are regions in weight space corresponding to different activation patterns.

Original: They are important for understanding neuron dynamics: roughly speaking, neurons in the same sector move in the same direction.
Revised: They are important for understanding neuron dynamics: roughly speaking, neurons in the same sector move in the same direction.

Original: Deﬁnition 1 (Sector) .
Revised: Deﬁnition 1 (Sector) .

Original: The sector associated to a sequence of signs σ ∈ {− 1 , 0 , 1 } n is the region in input space deﬁned as
Revised: The sector associated to a sequence of signs σ ∈ {− 1 , 0 , 1 } n is the region in input space deﬁned as

Original: [Equation]
Revised: [Equation]

Original: We may also refer to the sign sequence σ itself as a sector.
Revised: We may also refer to the sign sequence σ itself as a sector.

Original: Extremal sectors are sectors where neurons tend to cluster early in training.
Revised: Some sectors are attractors early in training, i.e. neurons tend to converge to them. Such attracting sectors are called extremal sectors .

Original: To characterise them formally, we ﬁrstneed to introduce the function G : S d 
Revised: To give a formal deﬁnition, we ﬁrst introduce the function G : S d − 1 → R ,

Original: [Equation]
Revised: [Equation]

Original: 1 → R ,
Revised: x i ) .

Original: [Equation]
Revised: Intuitively, (normalised) neurons early in training behave as if they were locally optimising G , they therefore tend to cluster around the local optima of G . We formally deﬁne extremal sectors as sectors containing these local optima.

Original: Deﬁnition 2 (Extremal directions and sectors) .
Revised: Deﬁnition 2 (Extremal directions and sectors) .

Original: We say that
Revised: We say that

Original: w ∈ S
Revised: w ∈ S

Original: d − 1 is a positive extremal direction , if it is a strict local maximum of G .
Revised: d − 1 is a positive extremal direction , if it is a strict local maximum of G .

Original: We say that w is a negative extremal direction if it is a strict local minimum of G .
Revised: We say that w is a negative extremal direction if it is a strict local minimum of G .

Original: A sector is called (positive/negative) extremal, if it contains a (positive/negative) extremal direction.
Revised: A sector is called (positive/negative) extremal, if it contains a (positive/negative) extremal direction.

Original: The following lemma (Maennel et al., 2018, Lemma 5) shows that all neurons either turn off, i.e. become deactivated for all training examples and stop updating, or converge to extremal sectors.
Revised: The following lemma (Maennel et al., 2018, Lemma 5) shows that all neurons either turn off, i.e. become deactivated for all training examples and stop updating, or converge to extremal sectors.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let a two-layer ReLU network f θ be balanced at initialisation and trained by gradient ﬂow.
Revised: Let a two-layer ReLU network f θ be balanced at initialisation and trained by gradient ﬂow.

Original: Assume that the loss derivative (cid:96) (cid:48) i is Lipschitz continuous.
Revised: Assume that the loss derivative (cid:96) (cid:48) i is Lipschitz continuous.

Original: Then, for almost all datasets and almost all initialisations with λ small enough, there exists a time T such that each neuron satisﬁes one of these three conditions:
Revised: Then, for almost all datasets and almost all initialisations with λ small enough, there exists a time T such that each neuron satisﬁes one of these three conditions:

Original: • w j ( T ) ∈ S σ where σ ≤ 0 and so w j remains constant for t ≥ T , or • a j ( T ) > 0 and
Revised: • w j ( T ) ∈ S σ where σ ≤ 0 and so w j remains constant for t ≥ T , or • a j ( T ) > 0 and

Original: w j ( T ) ∈ S σ where σ is a positive extremal sector, or • a j ( T ) < 0 and
Revised: w j ( T ) ∈ S σ where σ is a positive extremal sector, or • a j ( T ) < 0 and

Original: w j ( T ) ∈ S σ where σ is a negative extremal sector.
Revised: w j ( T ) ∈ S σ where σ is a negative extremal sector.

Original: 4. O RTHOGONAL SEPARABILITY : T WO ABSORBING EXTREMAL SECTORS
Revised: 4. O RTHOGONAL SEPARABILITY : T WO ABSORBING EXTREMAL SECTORS

Original: Lemma 1 shows that by the end of the early phase of training, neurons have converged to extremal sectors.
Revised: Lemma 1 shows that by the end of the early phase of training, neurons have converged to extremal sectors.

Original: Although eq.
Revised: Although eq.

Original: (11) shows that the number of extremal sectors depends only on the data (i.e. is independent of model expressivity), it is a priori unclear how many extremal sectors there are for a given dataset, or what happens once neurons have converged to extremal sectors.
Revised: (12) shows that the number of extremal sectors depends only on the data (i.e. is independent of model expressivity), it is a priori unclear how many extremal sectors there are for a given dataset, or what happens once neurons have converged to extremal sectors.

Original: We now answer both of these questions for orthogonally separable datasets.
Revised: We now answer both of these questions for orthogonally separable datasets.

Original: First, we claim that for orthogonally separable datasets, there are only two extremal sectors, one corresponding to the positive data subset and one corresponding to the negative data subset.
Revised: First, we claim that for orthogonally separable datasets, there are only two extremal sectors, one corresponding to the positive data subset and one corresponding to the negative data subset.

Original: That is, by converging to an extremal sector, neurons ‘choose’ whether to activate for positive examples or for negative examples.
Revised: That is, by converging to an extremal sector, neurons ‘choose’ whether to activate for positive examples or for negative examples.

Original: They thus naturally form two groups of similar neurons.
Revised: They thus naturally form two groups of similar neurons.

Original: Lemma 2.
Revised: Lemma 2.

Original: In the setting of Theorem 1, there is exactly one positive extremal direction and exactly one negative extremal direction.
Revised: In the setting of Theorem 1, there is exactly one positive extremal direction and exactly one negative extremal direction.

Original: The positive extremal sector σ + is given by
Revised: The positive extremal sector σ + is given by

Original: [Equation]
Revised: [Equation]

Original: and the negative extremal sector σ − is given by
Revised: and the negative extremal sector σ − is given by

Original: [Equation]
Revised: [Equation]

Original: [-, 0 , if y j = 1 and x (cid:124) j x i = 
-]
Revised: [-, 0 , if y j = 1 and x (cid:124) j x i = 
-]

Original: for all i with y i = − 1 .

Original: Second, we show that once a neuron reaches an extremal sector, it remains in the sector forever, i.e. its activation pattern remains ﬁxed for the rest of training.
Revised: Second, we show that once a neuron reaches an extremal sector, it remains in the sector forever, i.e. its activation pattern remains ﬁxed for the rest of training.

Original: Lemma 3.
Revised: Lemma 3.

Original: Assume the setting of Theorem 1.
Revised: Assume the setting of Theorem 1.

Original: If at time T the neuron ( a j , w j ) satisﬁes a j ( T ) >and w j ( T ) ∈ S σ , where σ is the positive extremal sector (eq. (12) ), then for t ≥ T , w j ( t ) ∈ S σ .
Revised: If at time T the neuron ( a j , w j ) satisﬁes a j ( T ) >and w j ( T ) ∈ S σ , where σ is the positive extremal sector (eq. (13) ), then for t ≥ T , w j ( t ) ∈ S σ .

Original: The same holds if a j ( T ) < 0 and σ is the negative extremal sector (eq.(13) ).
Revised: The same holds if a j ( T ) < 0 and σ is the negative extremal sector (eq.

Original: 4. P ROOF OF T HEOREM
Revised: 4. P ROOF OF T HEOREM

Original: Once neurons enter their respective absorbing sectors, the second phase of training begins.
Revised: Once neurons enter their respective absorbing sectors, the second phase of training begins.

Original: In this phase, the network’s activation patterns are ﬁxed: some neurons are active and update on the positive examples, while the others are active and update on the negative examples.
Revised: In this phase, the network’s activation patterns are ﬁxed: some neurons are active and update on the positive examples, while the others are active and update on the negative examples.

Original: The network thus behaves like an ensemble of independent linear subnetworks trained on subsets of the data.
Revised: The network thus behaves like an ensemble of independent linear subnetworks trained on subsets of the data.

Original: Once this happens, it becomes possible to apply existing results for linear networks; in particular, each subnetwork converges to its respective max-margin classiﬁer.
Revised: Once this happens, it becomes possible to apply existing results for linear networks; in particular, each subnetwork converges to its respective max-margin classiﬁer.

Original: We give more details in the proof below.
Revised: We give more details in the proof below.

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: By Lemmas 1 and 2, there exists a time T such that each neuron satisﬁes either
Revised: By Lemmas 1 and 2, there exists a time T such that each neuron satisﬁes either

Original: • w j ( T ) ∈ S σ where σ ≤ 0 and w j remains constant for t ≥ T , or • a j ( T ) > 0 and
Revised: • w j ( T ) ∈ S σ where σ ≤ 0 and w j remains constant for t ≥ T , or • a j ( T ) > 0 and

Original: w j ( T ) ∈ S σ + , or • a j ( T ) < 0 and w j ( T ) ∈ S σ − ,
Revised: w j ( T ) ∈ S σ + , or • a j ( T ) < 0 and w j ( T ) ∈ S σ − ,

Original: where σ + , σ − are the unique positive and negative extremal sectors given by eqs.
Revised: where σ + , σ − are the unique positive and negative extremal sectors given by eqs.

Original: (12) and (13).
Revised: (13) and (14).

Original: Denote by J 0 , J + , J − , the sets of neurons satisfying the ﬁrst, the second, and the third condition respectively.
Revised: Denote by J 0 , J + , J − , the sets of neurons satisfying the ﬁrst, the second, and the third condition respectively.

Original: By Lemma 3, if j ∈ J + then w j ( t ) ∈ S σ + for all t ≥ T , and if j ∈ J − then w j ( t ) ∈ S σ − for t ≥ T .
Revised: By Lemma 3, if j ∈ J + then w j ( t ) ∈ S σ + for all t ≥ T , and if j ∈ J − then w j ( t ) ∈ S σ − for t ≥ T .

Original: Hence, for t ≥ T , if x i is such that y i
Revised: Hence, for t ≥ T , if x i is such that y i

Original: = 1 then
Revised: = 1 then

Original: [Equation]
Revised: [Equation]

Original: Combined with Lemma A.3, this implies that for k ∈ J + ,
Revised: Combined with Lemma A.3, this implies that for k ∈ J + ,

Original: [Equation]
Revised: [Equation]

Original: (where we have used that P w x i
Revised: (where we have used that P w x i

Original: = x
Revised: = x

Original: i for i with y i = 1 due to positive extremality).
Revised: i for i with y i = 1 due to positive extremality).

Original: From eq.
Revised: From eq.

Original: (15) it follows that the evolution of neurons in J + depends only on positive examples and other neurons in J + .
Revised: (16) it follows that the evolution of neurons in J + depends only on positive examples and other neurons in J + .

Original: The neurons behave linearly on the positive data subset, while ignoring the negative subset.
Revised: The neurons behave linearly on the positive data subset, while ignoring the negative subset.

Original: The same argument shows that the evolution of neurons in J − depends only on other neurons in J − and the negative data subset, on which the neurons act linearly.
Revised: The same argument shows that the evolution of neurons in J − depends only on other neurons in J − and the negative data subset, on which the neurons act linearly.

Original: In other words, from time T onwards the
Revised: In other words, from time T onwards the

Original: ReLU network decomposes into a constant part and two independent linear networks, one trained on the positive data subset and the other trained on the negative data subset.
Revised: ReLU network decomposes into a constant part and two independent linear networks, one trained on the positive data subset and the other trained on the negative data subset.

Original: We can therefore apply existing max-margin convergence results for linear networks to each of the linear subnetworks.
Revised: We can therefore apply existing max-margin convergence results for linear networks to each of the linear subnetworks.

Original: Denote by W (cid:124) =
Revised: Denote by W (cid:124) =

Original: [ W (cid:124) 0 , W (cid:124)
Revised: [ W (cid:124) 0 , W (cid:124)

Original: + , W (cid:124) − ] the three parts of the weight matrix.
Revised: + , W (cid:124) − ] the three parts of the weight matrix.

Original: Then by (Ji & Telgarsky, 2019a, Theorems 2.2 and 2.8) and (Ji & Telgarsky, 2020, Theorem 3.1), there exist vectors ¯ u , ¯ z , such that
Revised: Then by (Ji & Telgarsky, 2019a, Theorems 2.2 and 2.8) and (Ji & Telgarsky, 2020, Theorem 3.1), there exist vectors ¯ u , ¯ z , such that

Original: [Equation]
Revised: [Equation]

Original:  (We allow ¯ u , ¯ z ∈ R 0 to account for the fact that J + , J − may be empty).
Revised: [Equation] (We allow ¯ u , ¯ z ∈ R 0 to account for the fact that J + , J − may be empty).

Original: We now need to relate (cid:107) W + (cid:107) F and (cid:107) W − (cid:107) F to (cid:107) W (cid:107) F .
Revised: We now need to relate (cid:107) W + [Equation] F and (cid:107) W − (cid:107) F to (cid:107) W (cid:107) F .

Original: In particular, it will be useful to show that (cid:107) W + ( t ) (cid:107) 2 F / log t has a limit as t → ∞ ; the same is true for (cid:107) W − ( t ) (cid:107) 2 F / log t (by the same argument).
Revised: In particular, it will be useful to show that (cid:107) W + ( t ) (cid:107) 2 F / log t has a limit as t → ∞ ; the same is true for (cid:107) W − ( t ) (cid:107) 2 F / log t (by the same argument).

Original: If J + or J − is empty, this is trivially true and the limit is 0.
Revised: If J + or J − is empty, this is trivially true and the limit is 0.

Original: Otherwise, consider the learning of the positive linear subnetwork, whose objective is effectively (cid:96) + ( θ ) :
Revised: Otherwise, consider the learning of the positive linear subnetwork, whose objective is effectively (cid:96) + ( θ ) :

Original: = (cid:80) i : y i =1 (cid:96) i ( f θ ( x i )) .
Revised: = (cid:80) i : y i =1 (cid:96) i ( f θ ( x i )) .

Original: By (Ji & Telgarsky, 2019a, Theorem 2.2), we know that (cid:96) + (  t ) → 0 as t → ∞ .
Revised: By (Ji & Telgarsky, 2019a, Theorem 2.2), we know that (cid:96) + ( θ ( t )) → 0 as t → ∞ .

Original: Following (Lyu & Li, 2020, Deﬁnition
Revised: Following (Lyu & Li, 2020, Deﬁnition

Original: A.3), deﬁne
Revised: A.3), deﬁne

Original: [Equation]
Revised: [Equation]

Original: 18) where g ( q ) := − log (exp(exp( − q )) − 1) for the cross-entropy loss.
Revised: where g ( q ) := − log (exp(exp( − q )) − 1) for the cross-entropy loss.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original:  Using the Taylor expansion exp( u ) =
Revised: [Equation] Using the Taylor expansion exp( u ) =

Original: 1 + Θ( u ) for u → 0 and (Lyu & Li, 2020, Corollary A.11), we obtain 
Revised: 1 + Θ( u ) for u → 0 and (Lyu & Li, 2020, Corollary A.11), we obtain [Equation]

Original: [Equation]
Revised: [Equation]

Original: By (Lyu & Li, 2020, Theorem A.7:1), ˜ γ is increasing in t and hence converges; it follows that (cid:107) W + ( t ) (cid:107) 2 F / log t has a limit.
Revised: By (Lyu & Li, 2020, Theorem A.7:1), ˜ γ is increasing in t and hence converges; it follows that (cid:107) W + ( t ) (cid:107) 2 F / log t has a limit.

Original: By (Lyu & Li, 2020, Corollary A.11), (cid:107) W + ( t ) (cid:107) 2 F = Θ(log t ) , implying that the limit is ﬁnite and strictly positive.
Revised: By (Lyu & Li, 2020, Corollary A.11), (cid:107) W + ( t ) (cid:107) 2 F = Θ(log t ) , implying that the limit is ﬁnite and strictly positive.

Original: We will denote it by ν
Revised: We will denote it by ν

Original: + and the analogous quantity for W − by ν − .
Revised: + and the analogous quantity for W − by ν − .

Original: We now return to the main thread of the proof.
Revised: We now return to the main thread of the proof.

Original: We analyse the convergence of W ( t ) / (cid:107) W ( t ) (cid:107) F by analysing W 0 / (cid:107) W ( t ) (cid:107) F , W + ( t ) / (cid:107) W ( t ) (cid:107) F and W − ( t ) / (cid:107) W ( t ) (cid:107) F in turn.
Revised: We analyse the convergence of W ( t ) / (cid:107) W ( t ) (cid:107) F by analysing W 0 / (cid:107) W ( t ) (cid:107) F , W + ( t ) / (cid:107) W ( t ) (cid:107) F and W − ( t ) / (cid:107) W ( t ) (cid:107) F in turn.

Original: Since (cid:107) W ( t ) (cid:107) 2 F = (cid:107) W 0 (cid:107) 2 F + (cid:107) W + ( t ) (cid:107) 2 F + (cid:107) W − ( t ) (cid:107) 2 F ,
Revised: Since (cid:107) W ( t ) (cid:107) 2 F = (cid:107) W 0 (cid:107) 2 F + (cid:107) W + ( t ) (cid:107) 2 F + (cid:107) W − ( t ) (cid:107) 2 F ,

Original: [Equation]
Revised: [Equation]

Original: Now observe that with probability at least 1 − 1 / 2 p over the random initialisation, ν + + ν − > 0 (or equivalently, J + ∪ J − (cid:54) = ∅ ).
Revised: Now observe that with probability at least 1 − 1 / 2 p over the random initialisation, ν + + ν − > 0 (or equivalently, J + ∪ J − (cid:54) = ∅ ).

Original: To prove this, let x i + be any training example with y i + = 1 and let x i − be any training example with y i − = − 1 .
Revised: To prove this, let x i + be any training example with y i + = 1 and let x i − be any training example with y i − = − 1 .

Original: Then by Lemma B.1, if a neuron ( a j , w j ) is initialised
Revised: Then by Lemma B.1, if a neuron ( a j , w j ) is initialised

Original: such that a j (0) > 0 and w j (0) (cid:124) x
Revised: such that a j (0) > 0 and w j (0) (cid:124) x

Original: i + > 0 then for t ≥ 0 , w j ( t ) (cid:124) x i + > 0 .
Revised: i + > 0 then for t ≥ 0 , w j ( t ) (cid:124) x i + > 0 .

Original: This holds in particular at time T .
Revised: This holds in particular at time T .

Original: The neuron j thus cannot be in J 0 nor J − , implying j ∈ J + .
Revised: The neuron j thus cannot be in J 0 nor J − , implying j ∈ J + .

Original: Similarly, if the neuron is initialised such that a j (0) < 0 and w j (0) (cid:124) x i − > 0 , then j ∈ J − .
Revised: Similarly, if the neuron is initialised such that a j (0) < 0 and w j (0) (cid:124) x i − > 0 , then j ∈ J − .

Original: The probability that one of the two initialisations occurs for a single neuron j is 1 / 2 , as P w j [ w j (0) (cid:124) x > 0 ]
Revised: The probability that one of the two initialisations occurs for a single neuron j is 1 / 2 , as P w j [ w j (0) (cid:124) x > 0 ]

Original: = 1 / 2 for any ﬁxed x .
Revised: = 1 / 2 for any ﬁxed x .

Original: Hence, the probability that j ∈ J 0 is at most 1 − 1 / 2 = 1 / 2 , and the probability that [ p ] ⊆
Revised: Hence, the probability that j ∈ J 0 is at most 1 − 1 / 2 = 1 / 2 , and the probability that [ p ] ⊆

Original: J 0 is at most 1 / 2 p .
Revised: J 0 is at most 1 / 2 p .

Original: It follows that with probability at least 1 − 1 / 2 p , 
Revised: It follows that with probability at least 1 − 1 / 2 p , [Equation]

Original: [Equation]
Revised: [Equation]

Original: Also, by eqs.
Revised: Also, by eqs.

Original: (16) and (21),
Revised: (17) and (22),

Original: [Equation]
Revised: [Equation]

Original: and similarly
Revised: and similarly

Original: [Equation]
Revised: [Equation]

Revised: For j ∈ J + and t ≥ T we moreover know that if y i = 1 then w j ( t ) (cid:124) x i > 0 because w j ( t ) ∈ S σ + .

Revised: As the same property holds for w + , it follows that ¯ u j ≥ 0 .

Revised: By a similar argument, ¯ z j ≥ 0 .

Original: Combining the last three equations yields the result.
Revised: Combining the last three equations then proves eq. As for eq.

Original: E XPERIMENTS
Revised: (9), we know by Lemma A.4 that a j ( t ) = s j (cid:107) w j ( t ) (cid:107) for some s j ∈ {± 1 } , implying (cid:107) a [Equation] t )

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: W

Revised: [Equation]

Revised: t ) (cid:107) F .

Revised: Hence, for j ∈ J + ,

Revised: [Equation]

Revised: 26) by eq.

Revised: (24), where u j ≥ 0 .

Revised: For j ∈ J + we also know that a j ( t ) ≥ 0 , so s j = 1 and

Revised: [Equation]

Revised: By a similar argument, we obtain that for j ∈ J − ,

Revised: [Equation]

Original:  In this section, we ﬁrst verify that the theoretical result (Theorem 1) is predictive of experimental outcomes, even when some technical assumptions are violated.
Revised: Finally, for j ∈ J 0 , a j ( t ) is constant and so [Equation]  E XPERIMENTS In this section, we ﬁrst verify that the theoretical result (Theorem 1) is predictive of experimental outcomes, even when some technical assumptions are violated.

Original: Second, we present evidence that a similar result may hold for deeper networks as well, although this goes beyond Theorem 1.
Revised: Second, we present evidence that a similar result may hold for deeper networks as well, although this goes beyond Theorem 1.

Original: 5. T WO - LAYER NETWORKS
Revised: 5. T WO - LAYER NETWORKS

Original: To see how well the theory holds up, we train a two-layer ReLU network with 100 neurons on a synthetic orthogonally separable dataset consisting of 500 examples in R 20 .
Revised: To see how well the theory holds up, we train a two-layer ReLU network with 100 neurons on a synthetic orthogonally separable dataset consisting of 500 examples in R 20 .

Original: The dataset is constructed from an iid Gaussian dataset by ﬁltering, to ensure orthogonal separability and w + (cid:54)≈ − w − (for visualisation purposes).
Revised: The dataset is constructed from an iid Gaussian dataset by ﬁltering, to ensure orthogonal separability and w + (cid:54)≈ − w − (for visualisation purposes).

Original: Speciﬁcally, let z :=
Revised: Speciﬁcally, let z :=

Original: [1 , − 1 , . . .
Revised: [1 , − 1 , . . .

Original: , 1 , − 1] .
Revised: , 1 , − 1] .

Original: A Gaussian-sampled point x is included with label +1 if it lies in the ﬁrst orthant and x (cid:124)
Revised: A Gaussian-sampled point x is included with label +1 if it lies in the ﬁrst orthant and x (cid:124)

Original: z ≥ 0 , included with label − 1 if it lies in the orthant opposite to the ﬁrst and x (cid:124)
Revised: z ≥ 0 , included with label − 1 if it lies in the orthant opposite to the ﬁrst and x (cid:124)

Original: z ≥ 0 , and discarded otherwise.
Revised: z ≥ 0 , and discarded otherwise.

Original: We train by stochastic gradient descent with batch size 50 and a learning rate of 0 .
Revised: We train by stochastic gradient descent with batch size 50 and a learning rate of 0 .

Original: 1 for 500 epochs.
Revised: 1 for 500 epochs.

Original: At initialisation, we multiply all weights by 0 .
Revised: At initialisation, we multiply all weights by 0 .

Original: This reﬂects a setting where both key assumptions of Theorem 1 – orthogonal separability and small initialisation – hold, while the other assumptions are relaxed to approach real-life practice.
Revised: This reﬂects a setting where both key assumptions of Theorem 1 – orthogonal separability and small initialisation – hold, while the other assumptions are relaxed to approach real-life practice.

Original: Figure 1 shows the results.
Revised: Figure 1 shows the results.

Original: Figure 1a shows the top 10 singular values of the ﬁrst-layer weight matrix W ∈ R 100 × 20 after training.
Revised: Figure 1a shows the top 10 singular values of the ﬁrst-layer weight matrix W ∈ R 100 × 20 after training.

Original: We see that despite its size, the matrix has rank only two: all singular values except the ﬁrst two are effectively zero.
Revised: We see that despite its size, the matrix has rank only two: all singular values except the ﬁrst two are effectively zero.

Original: This is exactly as predicted by the theorem.
Revised: This is exactly as predicted by the theorem.

Original: Furthermore, when we project the neurons on the positive-variance dimensions (Figure 1b), we see that they align along two main directions.
Revised: Furthermore, when we project the neurons on the positive-variance dimensions (Figure 1b), we see that they align along two main directions.

Original: To see how well these directions align with the predicted max-margin directions, we compute the correlation (normalised inner product) of each neuron with its respective max-margin direction.
Revised: To see how well these directions align with the predicted max-margin directions, we compute the correlation (normalised inner product) of each neuron with its respective max-margin direction.

Original: Figure 1c shows the histogram of these correlations.
Revised: Figure 1c shows the histogram of these correlations.

Original: We see that the correlation is generally high, above 0.9 for most neurons.
Revised: We see that the correlation is generally high, above 0.9 for most neurons.

Original: Overall we ﬁnd very good agreement with theory.
Revised: Overall we ﬁnd very good agreement with theory.

Original: [Figure]
Revised: [Figure]

Original: a) Top 10 sing.values of W b) Projected neurons

Original: [Table]

Original: [Figure]

Original: 5. D EEPER NETWORKS
Revised: 5. D EEPER NETWORKS

Original: We now explore the behaviour of deeper networks on orthogonally separable data.
Revised: We now explore the behaviour of deeper networks on orthogonally separable data.

Original: We train a residual network rather than a fully-connected one.
Revised: We train a residual network rather than a fully-connected one.

Original: The reason for this is that fully-connected networks with small initialisation are hard to train: early in training, the gradients are vanishingly small but then grow very quickly.
Revised: The reason for this is that fully-connected networks with small initialisation are hard to train: early in training, the gradients are vanishingly small but then grow very quickly.

Original: We therefore found setting a numerically stable learning rate rather delicate.
Revised: We therefore found setting a numerically stable learning rate rather delicate.

Original: We consider a residual network f θ : R d → R parameterised by θ (cid:44) { W 1 , . . .
Revised: We consider a residual network f θ : R d → R parameterised by θ (cid:44) { W 1 , . . .

Original: , W L } , of the form
Revised: , W L } , of the form

Original: [Equation]
Revised: [Equation]

Original: for l ∈ [2 , L− 1]
Revised: for l ∈ [2 , L

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where p is the network’s width, and W 1 ∈ R p × d , W l ∈ R p × p and W L ∈ R 1 × p are its weights.
Revised: (30) where p is the network’s width, and W 1 ∈ R p × d , W l ∈ R p × p and W L ∈ R 1 × p are its weights.

Original: We train such a four-layer residual net with width 100 on the same dataset and using the same optimiser and hyper-parameters as in Section 5.1.
Revised: We train such a four-layer residual net with width 100 on the same dataset and using the same optimiser and hyper-parameters as in Section 5.1.

Original: Figure 2 shows the results.
Revised: Figure 2 shows the results.

Original: The results are very similar to what we observe for two-layer nets: the weight matrices are all rank two (Figure 2a-c), and the weight matrices’ rows align in two main directions (Figure 2d-f).
Revised: The results are very similar to what we observe for two-layer nets: the weight matrices are all rank two (Figure 2a-c), and the weight matrices’ rows align in two main directions (Figure 2d-f).

Original: It is unclear what these directions are for the intermediate layers of the network, but for the ﬁrst layer, we conjecture it is again the max-margin directions, as suggested by Figure 2g.[Figure]
Revised: It is unclear what these directions are for the intermediate layers of the network, but for the ﬁrst layer, we conjecture it is again the max-margin directions, as suggested by Figure 2g.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: There is a lot of prior work on the implicit bias of gradient descent for various linear models.
Revised: There is a lot of prior work on the implicit bias of gradient descent for various linear models.

Original: For logistic regression, Soudry et al.
Revised: For logistic regression, Soudry et al.

Original: (2018) show that assuming an exponentially-tailed loss and linearly separable data, the normalised weight vector converges to the max-margin direction.
Revised: (2018) show that assuming an exponentially-tailed loss and linearly separable data, the normalised weight vector converges to the max-margin direction.

Original: Ji &
Revised: Ji &

Original: Telgarsky (2019b) extend this result to non-separable data, Nacson et al.
Revised: Telgarsky (2019b) extend this result to non-separable data, Nacson et al.

Original: (2019) extend it to superpolynomially-tailed losses, and Gunasekar et al.
Revised: (2019) extend it to superpolynomially-tailed losses, and Gunasekar et al.

Original: (2018a) considers different optimisation algorithms.
Revised: (2018a) considers different optimisation algorithms.

Original: For deep linear networks, Ji & Telgarsky (2019a) show that the end-to-end weight matrix converges to the max-margin solution and consecutive weight matrices align.
Revised: For deep linear networks, Ji & Telgarsky (2019a) show that the end-to-end weight matrix converges to the max-margin solution and consecutive weight matrices align.

Original: Gunasekar et al.
Revised: Gunasekar et al.

Original: (2018b) consider linear convolutional nets and prove convergence to a predictor related to the (cid:96) 2 /L bridge penalty.
Revised: (2018b) consider linear convolutional nets and prove convergence to a predictor related to the (cid:96) 2 /L bridge penalty.

Original: A few papers have started addressing the implicit bias problem for nonlinear (homogeneous or
Revised: A few papers have started addressing the implicit bias problem for nonlinear (homogeneous or

Original: ReLU) networks.
Revised: ReLU) networks.

Original: The problem is much harder and hence requires stronger assumptions.
Revised: The problem is much harder and hence requires stronger assumptions.

Original: Lyu &
Revised: Lyu &

Original: Li (2020) and Ji & Telgarsky (2020) assume that at some point during training, the network attains perfect classiﬁcation accuracy.
Revised: Li (2020) and Ji & Telgarsky (2020) assume that at some point during training, the network attains perfect classiﬁcation accuracy.

Original: Training from this point onward, Ji & Telgarsky (2020) show that the  network parameters converge in direction.
Revised: Training from this point onward, Ji & Telgarsky (2020) show that the [Figure] network parameters converge in direction.

Original: Lyu & Li (2020) show that this direction is a critical KKT point of the (nonlinear) max-margin problem.
Revised: Lyu & Li (2020) show that this direction is a critical KKT point of the (nonlinear) max-margin problem.

Original: A complementary approach is taken by Maennel et al.
Revised: A complementary approach is taken by Maennel et al.

Original: (2018) who analyse the very early phase of training, when the weights are close to the origin.
Revised: (2018) who analyse the very early phase of training, when the weights are close to the origin.

Original: For two-layer networks, they show convergence of neurons to extremal sectors.
Revised: For two-layer networks, they show convergence of neurons to extremal sectors.

Original: Our work can be seen as a ﬁrst step towards bridging the very early and the very late phase of training.
Revised: Our work can be seen as a ﬁrst step towards bridging the very early and the very late phase of training.

Original: Zooming out a bit, there is also work motivated by similar questions, but taking a different approach.
Revised: Zooming out a bit, there is also work motivated by similar questions, but taking a different approach.

Original: For example, Li & Liang (2018) show that two-layer ReLU nets trained on structured data converge to a solution that generalises well.
Revised: For example, Li & Liang (2018) show that two-layer ReLU nets trained on structured data converge to a solution that generalises well.

Original: Like ours, their analysis requires that the network’s activation patterns change little, but they achieve it by containing training in the neighbourhood of the (relatively large) initialisation (this is the standard lazy training argument Chizat et al.
Revised: Like ours, their analysis requires that the network’s activation patterns change little, but they achieve it by containing training in the neighbourhood of the (relatively large) initialisation (this is the standard lazy training argument Chizat et al.

Original: In contrast, we initialise much closer to zero, allowing the neurons to move more.
Revised: In contrast, we initialise much closer to zero, allowing the neurons to move more.

Original: Another related paper is Chizat & Bach (2020).
Revised: Another related paper is Chizat & Bach (2020).

Original: Using a mean-ﬁeld analysis, the authors show that inﬁnite-width two-layer ReLU nets converge to max-margin classiﬁers in a certain non-Hilbertian function space.
Revised: Using a mean-ﬁeld analysis, the authors show that inﬁnite-width two-layer ReLU nets converge to max-margin classiﬁers in a certain non-Hilbertian function space.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we prove that two-layer ReLU nets trained by gradient ﬂow on orthogonally separable data converge to a combination of the positive and the negative max-margin classiﬁer.
Revised: In this work, we prove that two-layer ReLU nets trained by gradient ﬂow on orthogonally separable data converge to a combination of the positive and the negative max-margin classiﬁer.

Original: To our knowledge, this is the ﬁrst result characterising the inductive bias of training neural networks with
Revised: To our knowledge, this is the ﬁrst result characterising the inductive bias of training neural networks with

Original: ReLU nonlinearities, that does not require inﬁnite width or huge overparameterisation.
Revised: ReLU nonlinearities, that does not require inﬁnite width or huge overparameterisation.

Original: The proof rests on a distinction between two phases of learning: an early phase, in which neurons specialise, and a late phase, in which the network’s activation pattern is ﬁxed and hence it behaves like an ensemble of linear subnetworks.
Revised: The proof rests on a distinction between two phases of learning: an early phase, in which neurons specialise and a late phase, in which the network’s activation pattern is ﬁxed and hence it behaves like an ensemble of linear subnetworks.

Original: This approach enables us to understand nonlinear ReLU networks in terms of the much better understood linear networks.
Revised: This approach enables us to understand nonlinear ReLU networks in terms of the much better understood linear networks.

Original: Our hope is that a similar strategy will prove fruitful in the context of deeper networks and more complicated datasets as well.
Revised: Our hope is that a similar strategy will prove fruitful in the context of deeper networks and more complicated datasets as well

Original: A B ASIC LEMMAS
Revised: A B ASIC LEMMAS

Original: This section collects a few lemmas useful for proofs.
Revised: This section collects a few lemmas useful for proofs.

Original: We assume the same setting and notation as Sections 2 and 4.
Revised: We assume the same setting and notation as Sections 2 and 4.

Original: In addition, we denote by P w the orthogonal projection onto span { x i | w (cid:124) x i = 0 } ⊥ , and by g : R d → R d ,
Revised: In addition, we denote by P w the orthogonal projection onto span { x i | w (cid:124) x i = 0 } ⊥ , and by g : R d → R d ,

Original: [Equation]
Revised: [Equation]

Original: A. L EMMAS ABOUT SECTORS
Revised: A. L EMMAS ABOUT SECTORS

Original: The following lemma gives a necessary condition for a vector to be an extremal direction.
Revised: The following lemma gives a necessary condition for a vector to be an extremal direction.

Original: Lemma A.1.
Revised: Lemma A.1.

Original: If w ∈ S
Revised: If w ∈ S

Original: d − 1 is an extremal direction, then g ( w ) =
Revised: d − 1 is an extremal direction, then g ( w ) =

Original: C w for some constant C .
Revised: C w for some constant C .

Original: Proof.
Revised: Proof.

Original: Let ˆ w ∈ S
Revised: Let ˆ w ∈ S

Original: d − 1 be a positive extremal direction (the negative case is analogous), and let ˆ w ∈ S ˆ σ .
Revised: d − 1 be a positive extremal direction (the negative case is analogous), and let ˆ w ∈ S ˆ σ .

Original: A sector is called open if σ i (cid:54) = 0 for all i ∈ [ n ] .
Revised: A sector is called open if σ i (cid:54) = 0 for all i ∈ [ n ] .

Original: Denote by A (ˆ σ ) the set of all open sectors adjacent to ˆ σ ,
Revised: Denote by A (ˆ σ ) the set of all open sectors adjacent to ˆ σ ,

Original: [Equation]
Revised: [Equation]

Original: Since ˆ w is a local maximum of G and G is sector-wise linear, ˆ w maximises G when constrained to (the closure of) any adjacent sector, i.e. for any σ ∈ A (ˆ σ ) ,
Revised: Since ˆ w is a local maximum of G and G is sector-wise linear, ˆ w maximises G when constrained to (the closure of) any adjacent sector, i.e. for any σ ∈ A (ˆ σ ) ,

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: For w in the feasible region, G can be treated as a linear function with ∇ G ( w ) =
Revised: For w in the feasible region, G can be treated as a linear function with ∇ G ( w ) =

Original: g ( w σ ) where w σ is any vector such that w σ ∈ S σ .
Revised: g ( w σ ) where w σ is any vector such that w σ ∈ S σ .

Original: Hence, the necessary ﬁrst-order KKT conditions for the problem (28) are
Revised: Hence, the necessary ﬁrst-order KKT conditions for the problem (33) are

Original: [Equation]
Revised: [Equation]

Original: where λ i ≥ 0 for all i , but λ i (cid:54) = 0 requires that the corresponding constraint is tight, σ i ˆ w (cid:124) x i = 0 .
Revised: where λ i ≥ 0 for all i , but λ i (cid:54) = 0 requires that the corresponding constraint is tight, σ i ˆ w (cid:124) x i = 0 .

Original: It follows that P ˆ w λ i σ i x i = 0 .
Revised: It follows that P ˆ w λ i σ i x i = 0 .

Original: Multiplying eq.
Revised: Multiplying eq.

Original: (29) from the left by P ˆ w therefore yields
Revised: (34) from the left by P ˆ w therefore yields

Original: [Equation]
Revised: [Equation]

Original: By adjacency, σ i = ˆ σ i whenever ˆ σ i ∈ {± 1 } , so they can differ only when ˆ σ i = 0 , i.e. when
Revised: By adjacency, σ i = ˆ σ i whenever ˆ σ i ∈ {± 1 } , so they can differ only when ˆ σ i = 0 , i.e. when

Original: P ˆ w x i = 0 .
Revised: P ˆ w x i = 0 .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: The following lemma describes the local behaviour of the function G (deﬁned in eq.
Revised: The following lemma describes the local behaviour of the function G (deﬁned in eq.

Original: Lemma A.2.
Revised: Lemma A.2.

Original: For w ∈ S d − 1 and v ∈ R d , there exists (cid:15) max > 0 such that for (cid:15) ∈ [0 , (cid:15) max ] ,
Revised: For w ∈ S d − 1 and v ∈ R d , there exists (cid:15) max > 0 such that for (cid:15) ∈ [0 , (cid:15) max ] ,

Original: G
Revised: G

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let g be deﬁned as in eq.
Revised: Let g be deﬁned as in eq.

Original: (26); then
Revised: (31); then

Original: [Equation]
Revised: [Equation]

Original: We now analyse w (cid:124) g
Revised: We now analyse w (cid:124) g

Original: ( w + (cid:15) v ) and (cid:15) v (cid:124) g
Revised: ( w + (cid:15) v ) and (cid:15) v (cid:124) g

Original: ( w + (cid:15) v ) separately, starting with the former.
Revised: ( w + (cid:15) v ) separately, starting with the former.

Original: Denote
Revised: Denote

Original: I (cid:15)i := 1 { ( w + (cid:15) v ) (cid:124) x i > 0 } .
Revised: I (cid:15)i := 1 { ( w + (cid:15) v ) (cid:124) x i > 0 } .

Original: Then g ( w + (cid:15) v ) can be written as g ( w + (cid:15) v ) =
Revised: Then g ( w + (cid:15) v ) can be written as g ( w + (cid:15) v ) =

Original: − n (cid:88) i
Revised: − n (cid:88) i

Original: = (cid:96) (cid:48)
Revised: = (cid:96) (cid:48)

Original: i (0) · I (cid:15)i I 0 i P w x
Revised: i (0) · I (cid:15)i I 0 i P w x

Original: i − n (cid:88) i = (cid:96) (cid:48)
Revised: i − n (cid:88) i = (cid:96) (cid:48)

Original: i (0) · I (cid:15)i (1 − I 0 i )
Revised: i (0) · I (cid:15)i (1 − I 0 i )

Original: P w x i − n (cid:88) i = (cid:96) (cid:48)
Revised: P w x i − n (cid:88) i = (cid:96) (cid:48)

Original: i (0) · I (cid:15)i ( P w + (cid:15) v − P w ) x i .
Revised: i (0) · I (cid:15)i ( P w + (cid:15) v − P w ) x i .

Original: Deﬁne (cid:15) max := 1 2 max (cid:15)> 0 (cid:15), subject to: sign { ( w + (cid:15) v ) (cid:124) x i } sign { w (cid:124) x i } ≥ 0 ∀ i.
Revised: Deﬁne (cid:15) max := 1 2 max (cid:15)> 0 (cid:15), subject to: sign { ( w + (cid:15) v ) (cid:124) x i } sign { w (cid:124) x i } ≥ 0 ∀ i.

Original: For (cid:15) ∈ [0 , (cid:15) max ] , I 0 i = 1 implies I (cid:15)i = 1 , so the ﬁrst term in eq.
Revised: For (cid:15) ∈ [0 , (cid:15) max ] , I 0 i = 1 implies I (cid:15)i = 1 , so the ﬁrst term in eq.

Original: (42) equals g ( w ) .
Revised: (47) equals g ( w ) .

Original: Regarding the second term, I (cid:15)i (1 − I 0 i ) is nonzero only if I (cid:15)i = 1 , I 0 i = 0 .
Revised: Regarding the second term, I (cid:15)i (1 − I 0 i ) is nonzero only if I (cid:15)i = 1 , I 0 i = 0 .

Original: For (cid:15) ∈ [0 , (cid:15) max ] , this can only happen if ( w + (cid:15) v ) (cid:124) x i > 0 and w (cid:124) x i = 0 .
Revised: For (cid:15) ∈ [0 , (cid:15) max ] , this can only happen if ( w + (cid:15) v ) (cid:124) x i > 0 and w (cid:124) x i = 0 .

Original: In this scenario however, P w x i = 0 , so the second term in eq.
Revised: In this scenario however, P w x i = 0 , so the second term in eq.

Original: (42) is zero.
Revised: (47) is zero.

Original: Regarding the third term, as long as (cid:15) ∈ [0 , (cid:15) max ] , ( w + (cid:15) v ) (cid:124) x
Revised: Regarding the third term, as long as (cid:15) ∈ [0 , (cid:15) max ] , ( w + (cid:15) v ) (cid:124) x

Original: i = 0 implies w (cid:124) x
Revised: i = 0 implies w (cid:124) x

Original: i
Revised: i

Original: = 0 , so w ∈ span { x j | w (cid:124) x j = 0 } ⊥ ⊆ span { x j | ( w + (cid:15) v ) (cid:124) x j = 0 } ⊥ (36) and w (cid:124)
Revised: = 0 , so w ∈ span { x j | w (cid:124) x j = 0 } ⊥ ⊆ span { x j | ( w + (cid:15) v ) (cid:124) x j = 0 } ⊥ (41) and w (cid:124)

Original: ( P w + (cid:15) v − P w ) =
Revised: ( P w + (cid:15) v − P w ) =

Original: w (cid:124) − w (cid:124)
Revised: w (cid:124) − w (cid:124)

Original: = 

Revised: =

Original: (cid:124) .
Revised: (cid:124) .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Turning to (cid:15) v (cid:124) g
Revised: Turning to (cid:15) v (cid:124) g

Original: ( w + (cid:15) v )
Revised: ( w + (cid:15) v )

Original: , we have that
Revised: , we have that

Original: [Equation]
Revised: [Equation]

Original: 38) where
Revised: 43) where

Original: [Equation]
Revised: [Equation]

Original: Plugging eq.
Revised: Plugging eq.

Original: (37) and eq.
Revised: (42) and eq.

Original: (38) into eq. (33) yields the result.
Revised: (43) into eq. (38) yields the result.

Original: A. T RAINING DYNAMICS
Revised: A. T RAINING DYNAMICS

Original: In the following lemma, we prove a formula for the evolution of the parameters of a two-layer network trained by gradient ﬂow (eq.
Revised: In the following lemma, we prove a formula for the evolution of the parameters of a two-layer network trained by gradient ﬂow (eq.

Original: The formula has appeared in Maennel et al.
Revised: The formula has appeared in Maennel et al.

Original: (2018) before (but without a proof).
Revised: (2018) before (but without a proof).

Original: Lemma A.3.
Revised: Lemma A.3.

Original: Assume that the training inputs with the zero vector { x i } i ∪ { 0 } are in general position.
Revised: Assume that the training inputs with the zero vector { x i } i ∪ { 0 } are in general position.

Original: Then a two-layer ReLU network trained by gradient ﬂow on ( X , y ) satisﬁes for all j ∈ [ p ] and almost all t ≥ 0 ,
Revised: Then a two-layer ReLU network trained by gradient ﬂow on ( X , y ) satisﬁes for all j ∈ [ p ] and almost all t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Fix θ , and denote by Σ θ ∈ {− 1 , 0 , 1 } p × n the activation matrix for f θ , Σ θ [ j, i ] (cid:44) sign w (cid:124)
Revised: Fix θ , and denote by Σ θ ∈ {− 1 , 0 , 1 } p × n the activation matrix for f θ , Σ θ [ j, i ] (cid:44) sign w (cid:124)

Original: j x i .
Revised: j x i .

Original: Then for any sequence θ k
Revised: Then for any sequence θ k

Original: → θ such that {∇ (cid:96) ( θ k ) } exists and has a limit, 
Revised: → θ such that {∇ (cid:96) ( θ k ) } exists and has a limit, [Equation]

Original: [Equation]
Revised: [Equation]

Original: Conversely, for any Σ in the set above, there exists a sequence θ k
Revised: Conversely, for any Σ in the set above, there exists a sequence θ k

Original: → θ such that lim k →∞ Σ θ k = Σ .
Revised: → θ such that lim k →∞ Σ θ k = Σ .

Original: To see this, observe that each w j can be approached separately.
Revised: To see this, observe that each w j can be approached separately.

Original: Let A be the matrix whose rows are formed by those x i for which w (cid:124)
Revised: Let A be the matrix whose rows are formed by those x i for which w (cid:124)

Original: j x i = 0 .
Revised: j x i = 0 .

Original: Then by the general position of inputs, A is a wide full-rank matrix and Aw j = 0 .
Revised: Then by the general position of inputs, A is a wide full-rank matrix and Aw j = 0 .

Original: It follows that for any (cid:15) ,
Revised: It follows that for any (cid:15) ,

Original: Aw = (cid:15) has a solution, which can be chosen convergent to w j as (cid:15) →0 .
Revised: Aw = (cid:15) has a solution, which can be chosen convergent to w j as (cid:15) →

Original: We deduce that
Revised: We deduce that

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne g ( Σ ) coordinate-wise as
Revised: where we deﬁne g ( Σ ) coordinate-wise as

Original: [Equation]
Revised: [Equation]

Original: Since the value of g ( Σ )[ a j ] is independent of Σ , this proves eq.
Revised: Since the value of g ( Σ )[ a j ] is independent of Σ , this proves eq.

Original: The proof of eq.
Revised: The proof of eq.

Original: (41) is slightly more complicated, as we need to pin down a single member of ∂(cid:96)
Revised: (46) is slightly more complicated, as we need to pin down a single member of ∂(cid:96)

Original: ( θ ( t )) .
Revised: ( θ ( t )) .

Original: To do that, we recall a result by Davis et al.
Revised: To do that, we recall a result by Davis et al.

Original: (2020), who show that for a large class of deep learning scenarios (which includes ours), the objective (cid:96) admits a chain rule,
Revised: (2020), who show that for a large class of deep learning scenarios (which includes ours), the objective (cid:96) admits a chain rule,

Original: i.e.
Revised: i.e.

Revised: (cid:96)

Original: [Equation]
Revised: [Equation]

Original: for almost all t ≥ 0 , (45) where the right-hand side above should be interpreted as the only element of the set { h (cid:124) ∂ θ /∂t | h ∈ ∂(cid:96)
Revised: for almost all t ≥ 0 , (50) where the right-hand side above should be interpreted as the only element of the set { h (cid:124) ∂ θ /∂t | h ∈ ∂(cid:96)

Original: ( θ ( t )) } .
Revised: ( θ ( t )) } .

Original: For t such that both eq.
Revised: For t such that both eq.

Original: (4) and eq.
Revised: (4) and eq.

Original: (45) hold,
Revised: (50) hold,

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: (46) implying that
Revised: implying that

Original: [Equation]
Revised: [Equation]

Original: Suppose h 1 , h 2 satisfy both eq.
Revised: Suppose h 1 , h 2 satisfy both eq.

Original: (4) and eq.
Revised: (4) and eq.

Original: (47) (taking the role of ∂ θ /∂t ).
Revised: (52) (taking the role of ∂ θ /∂t ).

Original: Then
Revised: Then

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: It follows that ∂ θ
Revised: It follows that ∂ θ

Original: /∂t
Revised: /∂t

Original: is the unique member of both span { ∂(cid:96)
Revised: is the unique member of both span { ∂(cid:96)

Original: ( θ ( t ))
Revised: ( θ ( t ))

Original: − ∂(cid:96)
Revised: − ∂(cid:96)

Original: ( θ ( t ) } ⊥ and − ∂(cid:96)
Revised: ( θ ( t ) } ⊥ and − ∂(cid:96)

Original: ( θ ( t )) .
Revised: ( θ ( t )) .

Original: By eqs.
Revised: By eqs.

Original: (43) and (44),
Revised: (48) and (49),

Original: [Equation]
Revised: [Equation]

Original:  where ξ ij [ w j ] = x i and all other elements of ξ ij are zero.
Revised: [Equation] where ξ ij [ w j ] = x i and all other elements of ξ ij are zero.

Original: Since
Revised: Since

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we obtain that for all ( i, j ) with w (cid:124)
Revised: we obtain that for all ( i, j ) with w (cid:124)

Original: j x i = 0 , ∂ w j /∂t ⊥ x i .
Revised: j x i = 0 , ∂ w j /∂t ⊥ x i .

Original: In other words,
Revised: In other words,

Original: [Equation]
Revised: [Equation]

Original: 51) where the inclusion follows from ∂ θ /∂t ∈ − ∂(cid:96)
Revised: where the inclusion follows from ∂ θ /∂t ∈ − ∂(cid:96)

Original: ( θ ( t )) and eqs.
Revised: ( θ ( t )) and eqs.

Original: (43) and (44).
Revised: (48) and (49).

Original: Now observe that by deﬁnition, P w j x i = 0 for all ( i, j ) with Σ θ
Revised: Now observe that by deﬁnition, P w j x i = 0 for all ( i, j ) with Σ θ

Original: [ j, i ] = 0 , hence the set in eq.
Revised: [ j, i ] = 0 , hence the set in eq.

Original: (51) is a singleton whose only element equals eq.
Revised: (56) is a singleton whose only element equals eq.

Original: The following lemma shows that a balanced two-layer network remains balanced and neurons keep their signs.
Revised: The following lemma shows that a balanced two-layer network remains balanced and neurons keep their signs.

Original: Lemma A.4.
Revised: Lemma A.4.

Original: If a two-layer neural network is balanced at initialisation and trained by gradient ﬂow with a loss whose derivative is bounded, then for t ≥ 0 ,
Revised: If a two-layer neural network is balanced at initialisation and trained by gradient ﬂow with a loss whose derivative is bounded, then for t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Lemma A.3, for almost all t ≥ 0 ,
Revised: By Lemma A.3, for almost all t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: i.e. a j and w j grow equally fast.
Revised: i.e. a j and w j grow equally fast.

Original: Since | a j (0) | = (cid:107) w j (0) (cid:107) at initialisation, | a j ( t ) | = (cid:107) w j ( t ) (cid:107) throughout training.
Revised: Since | a j (0) | = (cid:107) w j (0) (cid:107) at initialisation, | a j ( t ) | = (cid:107) w j ( t ) (cid:107) throughout training.

Original: Next denote by B, V > 0 some scalars such that | (cid:96) (cid:48)
Revised: Next denote by B, V > 0 some scalars such that | (cid:96) (cid:48)

Original: i ( u ) | ≤ B for all i ∈ [ n ] and u ∈ R , and (cid:107) x i (cid:107) ≤ V for all i ∈ [ n ] .
Revised: i ( u ) | ≤ B for all i ∈ [ n ] and u ∈ R , and (cid:107) x i (cid:107) ≤ V for all i ∈ [ n ] .

Original: Then (cid:12)(cid:12) ∂a 2 j /∂t (cid:12)(cid:12) ≤ nBa 2 j V , or equivalently (cid:12)(cid:12) ∂ log a 2 j /∂t (cid:12)(cid:12) ≤ nBV .
Revised: Then (cid:12)(cid:12) ∂a 2 j /∂t (cid:12)(cid:12) ≤ nBa 2 j V , or equivalently (cid:12)(cid:12) ∂ log a 2 j /∂t (cid:12)(cid:12) ≤ nBV .

Original: It follows that a 2 j ( t ) lies between a 2 j (0) exp( − nBV t ) and a 2 j (0) exp( nBV t ) , and hence a j cannot cross zero in ﬁnite time, proving eq.
Revised: It follows that a 2 j ( t ) lies between a 2 j (0) exp( − nBV t ) and a 2 j (0) exp( nBV t ) , and hence a j cannot cross zero in ﬁnite time, proving eq.

Original: B P ROOFS OF MAIN RESULTS
Revised: B P ROOFS OF MAIN RESULTS

Original: Lemma 2.
Revised: Lemma 2.

Original: In the setting of Theorem 1, there is exactly one positive extremal direction and exactly one negative extremal direction.
Revised: In the setting of Theorem 1, there is exactly one positive extremal direction and exactly one negative extremal direction.

Original: The positive extremal sector σ + is given by 
Revised: The positive extremal sector σ + is given by σ + j =   1 , if y j = 1 , − 1 , if y j = − 1 and x (cid:124) j x i < 0 for some i with y i = 1 , 0 , if y j = − 1 and x (cid:124) j

Original: [Equation]
Revised: [Equation]

Original: and the negative extremal sector σ − is given by
Revised: and the negative extremal sector σ − is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We will prove the positive case; the negative case follows by inverting all labels.
Revised: We will prove the positive case; the negative case follows by inverting all labels.

Original: Because
Revised: Because

Original: G is a continuous function on a compact domain, it has a maximum.
Revised: G is a continuous function on a compact domain, it has a maximum.

Original: At least one maximum must moreover be strict, or otherwise G would have to be constant.
Revised: At least one maximum must moreover be strict, or otherwise G would have to be constant.

Original: This shows that a positive extremal direction exists; we now show there is no more than one such direction.
Revised: This shows that a positive extremal direction exists; we now show there is no more than one such direction.

Original: By Lemma A.1, there cannot be more than one extremal direction per sector; it therefore sufﬁces to show that no sector except one, σ + , admits a positive extremal direction.
Revised: By Lemma A.1, there cannot be more than one extremal direction per sector; it therefore sufﬁces to show that no sector except one, σ + , admits a positive extremal direction.

Original: We will show that if w ∈ S
Revised: We will show that if w ∈ S

Original: d − 1 lies in any sector other than σ + , then w is not positive extremal; in particular we show that G ( w ) can be locally increased.
Revised: d − 1 lies in any sector other than σ + , then w is not positive extremal; in particular we show that G ( w ) can be locally increased.

Original: Let σ (cid:54) =
Revised: Let σ (cid:54) =

Original: σ + and let w ∈ S
Revised: σ + and let w ∈ S

Original: σ ∩ S d − 1 .
Revised: σ ∩ S d − 1 .

Original: By Lemma A.2, for any v ∈ R d there exists (cid:15) max > 0 such that for (cid:15) ∈ (0 , (cid:15) max ] ,
Revised: By Lemma A.2, for any v ∈ R d there exists (cid:15) max > 0 such that for (cid:15) ∈ (0 , (cid:15) max ] ,

Original: [Equation]
Revised: [Equation]

Original:  where
Revised: 59) where

Original: [Equation]
Revised: [Equation]

Original:  We now analyse the different possible realisations of σ , and for each we ﬁnd v ∈ R d such that ( G ( w ) + (cid:15)α ) / (cid:107) w + (cid:15) v (cid:107) > G ( w ) for small (cid:15) .
Revised: [Equation] We now analyse the different possible realisations of σ , and for each we ﬁnd v ∈ R d such that ( G ( w ) + (cid:15)α ) / (cid:107) w + (cid:15) v (cid:107) > G ( w ) for small (cid:15) .

Original: Suppose ﬁrst that σ j = − 1 for some example with y j = 1 , or that σ j = 1 for some example with y j = − 1 .
Revised: Suppose ﬁrst that σ j = − 1 for some example with y j = 1 , or that σ j = 1 for some example with y j = − 1 .

Original: Then set v :=
Revised: Then set v :=

Original: y j x j / (cid:107) x j (cid:107) .
Revised: y j x j / (cid:107) x j (cid:107) .

Original: By orthogonal separability, we have that α ≥ 0 .
Revised: By orthogonal separability, we have that α ≥ 0 .

Original: Also, σ j (cid:44) sign w (cid:124) x
Revised: Also, σ j (cid:44) sign w (cid:124) x

Original: j = − y j implies w (cid:124) v < 0 , therefore (cid:107) w + (cid:15) v (cid:107) < (cid:107) w (cid:107)
Revised: j = − y j implies w (cid:124) v < 0 , therefore (cid:107) w + (cid:15) v (cid:107) < (cid:107) w (cid:107)

Original: =
Revised: =

Original: 1 for (cid:15) small enough.
Revised: 1 for (cid:15) small enough.

Original: It follows that ( G ( w ) + (cid:15)α ) / (cid:107) w + (cid:15) v (cid:107) >
Revised: It follows that ( G ( w ) + (cid:15)α ) / (cid:107) w + (cid:15) v (cid:107) >

Original: G ( w ) .
Revised: G ( w ) .

Original: Next suppose that σ j = 0 for some example with y j = 1 , and set v := x j / (cid:107) x j (cid:107) .
Revised: Next suppose that σ j = 0 for some example with y j = 1 , and set v := x j / (cid:107) x j (cid:107) .

Original: Then α > 0 , because each term in eq.
Revised: Then α > 0 , because each term in eq.

Original: (55) is non-negative by orthogonal separability, and the term corresponding to i =
Revised: (60) is non-negative by orthogonal separability, and the term corresponding to i =

Original: j is strictly positive:
Revised: j is strictly positive:

Original: [Equation]
Revised: [Equation]

Original: (cid:48)

Original: j (0) 1 { (cid:15) (cid:107) x j (cid:107) > 0 }(cid:107) x j (cid:107) > 0 .

Original: From σ j (cid:44) sign w (cid:124) x j = 

Revised: From σ j (cid:44) sign w (cid:124) x j =

Original: it further follows that (cid:107) w + (cid:15) v (cid:107) = √ 1 + (cid:15) 2 .
Revised: it further follows that (cid:107) w + (cid:15) v (cid:107) = √ 1 + (cid:15) 2 .

Original: Hence, ( G ( w ) + (cid:15)α ) / (cid:107) w + (cid:15) v (cid:107) =
Revised: Hence, ( G ( w ) + (cid:15)α ) / (cid:107) w + (cid:15) v (cid:107) =

Original: ( G ( w ) + (cid:15)α ) /
Revised: ( G ( w ) + (cid:15)α ) /

Original: (1 + O ( (cid:15) 2 )) , which strictly exceeds G ( w ) for (cid:15) small enough.
Revised: (1 + O ( (cid:15) 2 )) , which strictly exceeds G ( w ) for (cid:15) small enough.

Original: We have thus shown that if σ is positive extremal, then necessarily σ i = 1 for all examples with y i = 1 , and σ i ∈ { 0 , − 1 } for examples with y i = − 1 .
Revised: We have thus shown that if σ is positive extremal, then necessarily σ i = 1 for all examples with y i = 1 , and σ i ∈ { 0 , − 1 } for examples with y i = − 1 .

Original: Suppose now that σ j = 0 for an example with y j = − 1 that satisﬁes x (cid:124) j x k < 0 for some k with y k = 1 .
Revised: Suppose now that σ j = 0 for an example with y j = − 1 that satisﬁes x (cid:124) j x k < 0 for some k with y k = 1 .

Original: Taking v := − x j / (cid:107) x j (cid:107) will make α strictly positive, as the term corresponding to i
Revised: Taking v := − x j / (cid:107) x j (cid:107) will make α strictly positive, as the term corresponding to i

Original: = k in eq.
Revised: = k in eq.

Original: (55) will be strictly positive (this term’s indicator equals 1, as we know from the above that σ k = 1 ).
Revised: (60) will be strictly positive (this term’s indicator equals 1, as we know from the above that σ k = 1 ).

Original: Like in the previous paragraph, (cid:107) w + (cid:15) v (cid:107)
Revised: Like in the previous paragraph, (cid:107) w + (cid:15) v (cid:107)

Original: = 1 + O ( (cid:15) 2 ) , which sufﬁces to show G ( w ) is locally submaximal.
Revised: = 1 + O ( (cid:15) 2 ) , which sufﬁces to show G ( w ) is locally submaximal.

Original: Finally, let x j be such that y j = − 1 and x (cid:124)
Revised: Finally, let x j be such that y j = − 1 and x (cid:124)

Original: j x i = 0 for all i with y i
Revised: j x i = 0 for all i with y i

Original: = 1 , and suppose that σ j = − 1 .
Revised: = 1 , and suppose that σ j = − 1 .

Original: With v :
Revised: With v :

Original: = P w x j / (cid:107) P w x j (cid:107) (we know that P w x j (cid:54)
Revised: = P w x j / (cid:107) P w x j (cid:107) (we know that P w x j (cid:54)

Original: = 0 because w (cid:124) x
Revised: = 0 because w (cid:124) x

Original: j (cid:54) = 0 by σ j = − 1 ), we have [Equation]
Revised: j (cid:54) = 0 by σ j = − 1 ), we have α = − (cid:107) P w x j (cid:107) n (cid:88) i = (cid:96) (cid:48)

Original: [Equation]
Revised: [Equation]

Original: We claim that each term in eq.
Revised: We claim that each term in eq.

Original: (57) is zero:
Revised: (62) is zero:

Original: For terms with σ i = − 1 , the indicator 1 { ( w + (cid:15) v ) (cid:124) x i > 0 } is zero.
Revised: For terms with σ i = − 1 , the indicator 1 { ( w + (cid:15) v ) (cid:124) x i > 0 } is zero.

Original: For terms with σ i = 0 , P w x i = 0 .
Revised: For terms with σ i = 0 , P w x i = 0 .

Original: (Also notice that such terms necessarily satisfy y i = − 1 and x (cid:124)
Revised: (Also notice that such terms necessarily satisfy y i = − 1 and x (cid:124)

Original: i x k = 0 for all k with y k = 1 , which we will need shortly.)
Revised: i x k = 0 for all k with y k = 1 , which we will need shortly.)

Original: Lastly, for terms with σ i = 1 , we know y i
Revised: Lastly, for terms with σ i = 1 , we know y i

Original: = 1 , and hence x (cid:124) i x l = 0 for all l with σ l = 0 .
Revised: = 1 , and hence x (cid:124) i x l = 0 for all l with σ l = 0 .

Original: In other words, x i ⊥ span { x l | w (cid:124) x l = 0 } , implying P w x i = x i .
Revised: In other words, x i ⊥ span { x l | w (cid:124) x l = 0 } , implying P w x i = x i .

Original: In the context of eq. (57), we obtain x (cid:124)
Revised: In the context of eq. (62), we obtain x (cid:124)

Original: j P w x
Revised: j P w x

Original: i
Revised: i

Original: = x (cid:124)
Revised: = x (cid:124)

Original: j x i = 0 , concluding the proof that α = 0 .
Revised: j x i = 0 , concluding the proof that α = 0 .

Original: Since σ j = − 1 , (cid:107) w + (cid:15) v (cid:107) <
Revised: Since σ j = − 1 , (cid:107) w + (cid:15) v (cid:107) <

Original: (cid:107) w (cid:107) =
Revised: (cid:107) w (cid:107) =

Original: 1 for small enough (cid:15) , and ( G ( w ) + (cid:15)α ) / (cid:107) w + (cid:15) v (cid:107) > G ( w ) .
Revised: 1 for small enough (cid:15) , and ( G ( w ) + (cid:15)α ) / (cid:107) w + (cid:15) v (cid:107) > G ( w ) .

Original: We have thus ruled out all sectors except σ + , proving that for orthogonally separable datasets there is a unique positive extremal sector.
Revised: We have thus ruled out all sectors except σ + , proving that for orthogonally separable datasets there is a unique positive extremal sector.

Original: Lemma 3.
Revised: Lemma 3.

Original: Assume the setting of Theorem 1.
Revised: Assume the setting of Theorem 1.

Original: If at time T the neuron ( a j , w j ) satisﬁes a j ( T ) >and w j ( T ) ∈ S σ , where σ is the positive extremal sector (eq. (12) ), then for t ≥ T , w j ( t ) ∈ S σ .
Revised: If at time T the neuron ( a j , w j ) satisﬁes a j ( T ) >and w j ( T ) ∈ S σ , where σ is the positive extremal sector (eq. (13) ), then for t ≥ T , w j ( t ) ∈ S σ .

Original: The same holds if a j ( T ) < 0 and σ is the negative extremal sector (eq.(13) ).
Revised: The same holds if a j ( T ) < 0 and σ is the negative extremal sector (eq.

Original: Proof.
Revised: Proof.

Original: We omit the neuron index, and only prove the positive case; the negative case is analogous.
Revised: We omit the neuron index, and only prove the positive case; the negative case is analogous.

Original: Denote σ ( t ) :
Revised: Denote σ ( t ) :

Original: = sign ( X (cid:124) w ( t )) .
Revised: = sign ( X (cid:124) w ( t )) .

Original: We proceed by contradiction.
Revised: We proceed by contradiction.

Original: Suppose there exists a time T 1 > T such that σ ( T 1 ) (cid:54) =
Revised: Suppose there exists a time T 1 > T such that σ ( T 1 ) (cid:54) =

Original: σ ( T ) .
Revised: σ ( T ) .

Original: Wlog, take T 1 such that σ ( t ) is constant on ( T, T 1 ) and denote this constant sector ¯ σ ; by continuity ¯ σ k = σ k
Revised: Wlog, take T 1 such that σ ( t ) is constant on ( T, T 1 ) and denote this constant sector ¯ σ ; by continuity ¯ σ k = σ k

Original: ( T ) if σ k ( T ) (cid:54) = 0 .
Revised: ( T ) if σ k ( T ) (cid:54) = 0 .

Original: Now consider σ k ( T ) = 0 .
Revised: Now consider σ k ( T ) = 0 .

Original: By the gradient ﬂow differential inclusion, for almost all t ∈ ( T, T 1 ) ,
Revised: By the gradient ﬂow differential inclusion, for almost all t ∈ ( T, T 1 ) ,

Original: [Equation]
Revised: [Equation]

Original: where each σ (cid:48) in the deﬁnition of the convex hull satisﬁes σ (cid:48)
Revised: where each σ (cid:48) in the deﬁnition of the convex hull satisﬁes σ (cid:48)

Original: i =
Revised: i =

Original: σ i ( T ) if σ i ( T ) (cid:54) = 0 , implying { i | 
Revised: σ i ( T ) if σ i ( T ) (cid:54) = 0 , implying { i | σ (cid:48) i = 1 } ⊆

Original: [Equation]
Revised: { i | σ i ( T ) = 1 } ∪ { i | σ i ( T ) = 0 } = { i | y i = 1 } ∪ { i | y i = − 1 and x (cid:124) i x j = 0 for all j with y j = 1 } .

Original: Denote the two sets in the last expression I + and I 0 , and consider the gradient corresponding to some σ (cid:48) in eq.
Revised: Denote the two sets in the last expression I + and I 0 , and consider the gradient corresponding to some σ (cid:48) in eq.

Original: The gradient terms corresponding to i ∈ I + are zero (because k ∈ I 

Revised: The gradient terms corresponding to i ∈ I + are zero (because k ∈ I

Original: and so x (cid:124)
Revised: and so x (cid:124)

Original: i x k = 0 ) and the terms corresponding to i ∈
Revised: i x k = 0 ) and the terms corresponding to i ∈

Original: I 

Revised: I

Original: (if there are any) are negative.
Revised: (if there are any) are negative.

Original: The total gradient for σ (cid:48) is therefore non-positive, which is preserved under taking convex hulls, and so we obtain ∂∂t w (cid:124) x k ≤ 0 .
Revised: The total gradient for σ (cid:48) is therefore non-positive, which is preserved under taking convex hulls, and so we obtain [Equation] x k ≤ 0 .

Original: It follows that ¯ σ k (cid:54) = 1 .
Revised: It follows that ¯ σ k (cid:54) = 1 .

Original: By Lemma A.3, for almost all t ∈ ( T [Equation] T 1 ) and any k ∈ [ n ] ,
Revised: By Lemma A.3, for almost all t ∈ ( T, T 1 ) and any k ∈ [ n ] ,

Original: [Equation]
Revised: [Equation]

Original:  where 1 { ¯ σ i = 1 } = 1 { y i = 1 } as we have shown above.
Revised: 65) where 1 { ¯ σ i = 1 } = 1 { y i = 1 } as we have shown above.

Original: Observe that for x i with y i = 1 , we have
Revised: Observe that for x i with y i = 1 , we have

Original: P w x i
Revised: P w x i

Original: = x i .
Revised: = x i .

Original: This is because P w projects onto
Revised: This is because P w projects onto

Original: [Equation]
Revised: [Equation]

Original: and x i lies in the right-hand side by the deﬁnition of positive extremal sector (eq. 
Revised: and x i lies in the right-hand side by the deﬁnition of positive extremal sector (eq. (13)).

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original:  One can easily check that if σ k ( T ) = 1 then ∂∂t w (cid:124) x k > 0 , if σ k ( T ) =
Revised: [Equation] One can easily check that if σ k ( T ) = 1 then ∂∂t w (cid:124) x k > 0 , if σ k ( T ) =

Original: − 1 then ∂∂t w (cid:124) x k < 0 , and if σ k ( T ) = 0 then ∂∂t w (cid:124) x k = 0 .
Revised: − 1 then ∂∂t w (cid:124) x k < 0 , and if σ k ( T ) = 0 then ∂∂t w (cid:124) x k = 0 .

Original: It follows that σ ( T 1 ) =
Revised: It follows that σ ( T 1 ) =

Original: σ ( T ) , which is a contradiction.
Revised: σ ( T ) , which is a contradiction.

Original: Lemma B.1.
Revised: Lemma B.1.

Original: Assume the setting of Theorem 1.
Revised: Assume the setting of Theorem 1.

Original: If at time T the neuron ( a j , w j ) satisﬁes a j ( T ) >and w j ( T ) (cid:124) x k > 0 for some k ∈ [ n ] with y k
Revised: If at time T the neuron ( a j , w j ) satisﬁes a j ( T ) >and w j ( T ) (cid:124) x k > 0 for some k ∈ [ n ] with y k

Original: = 1 , then for t ≥ T , w j ( t ) (cid:124) x k > 0 .
Revised: = 1 , then for t ≥ T , w j ( t ) (cid:124) x k > 0 .

Original: The same holds if instead a j ( T ) < 0 and y k
Revised: The same holds if instead a j ( T ) < 0 and y k

Original: = − 1 .
Revised: = − 1 .

Original: Proof.
Revised: Proof.

Original: We omit the neuron index, and only prove the positive case; the negative case is analogous.
Revised: We omit the neuron index, and only prove the positive case; the negative case is analogous.

Original: We will show that for almost all t ∈ ( T, ∞ ) , ∂ w (cid:124) x k /∂t ≥ 0 .
Revised: We will show that for almost all t ∈ ( T, ∞ ) , ∂ w (cid:124) x k /∂t ≥ 0 .

Original: By the gradient ﬂow differential inclusion, for almost all t ∈ ( T, ∞ ) ,
Revised: By the gradient ﬂow differential inclusion, for almost all t ∈ ( T, ∞ ) ,

Original: [Equation]
Revised: [Equation]

Original: Fix any σ (cid:48) and consider the summand corresponding to example i .
Revised: Fix any σ (cid:48) and consider the summand corresponding to example i .

Original: If y i = 1 , then − (cid:96) (cid:48) i ( t ) > 0 and x (cid:124) i x k > 0 , so the summand is non-negative.
Revised: If y i = 1 , then − (cid:96) (cid:48) i ( t ) > 0 and x (cid:124) i x k > 0 , so the summand is non-negative.

Original: If y i = − 1 , then − (cid:96) (cid:48) i ( t ) < 0 and x (cid:124) i x k ≤ 0 , so the summand is again non-negative.
Revised: If y i = − 1 , then − (cid:96) (cid:48) i ( t ) < 0 and x (cid:124) i x k ≤ 0 , so the summand is again non-negative.

Original: It follows that the sum is non-negative irrespective of σ (cid:48) , hence ∂ w (cid:124) x k /∂t ≥ 0 .
Revised: It follows that the sum is non-negative irrespective of σ (cid:48) , hence ∂ w (cid:124) x k /∂t ≥ 0 .

Original: Corollary 1.
Revised: Corollary 1.

Original: Under the conditions of Theorem 1, there exist constants u, z ≥ 0 such that
Revised: Under the conditions of Theorem 1, there exist constants u, z ≥ 0 such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Lemma A.4, (cid:107) θ (cid:107) 2 = (cid:107) a (cid:107) 

Revised: By Lemma A.4, (cid:107) θ (cid:107) 2 = (cid:107) a (cid:107)

Original: +
Revised: +

Original: (cid:107) W (cid:107) 2 F = 2 (cid:107) a (cid:107) 2 = 2 (cid:107) W (cid:107) 2 F .
Revised: (cid:107) W (cid:107) 2 F = 2 (cid:107) a (cid:107) 2 = 2 (cid:107) W (cid:107) 2 F .

Original: Then for any x ∈ R d ,
Revised: Then for any x ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: Denote ˜ a :=
Revised: Denote ˜ a :=

Original: lim t →∞ a ( t ) / (cid:107) a ( t ) (cid:107) ; this limit exists by (Ji & Telgarsky, 2020, Theorem 3.1).
Revised: lim t →∞ a ( t ) / (cid:107) a ( t ) (cid:107) .

Original: Then by Theorem 1, as t → ∞ ,
Revised: Then by Theorem 1, as t → ∞ ,

Original: [Equation] 
Revised: [Equation] [Equation]

Original: where in eq.
Revised: where in eq.

Original: (66) we used the fact that for all i ∈ [ p ] , either u i = 0 or z i = 0 , and in eq. 
Revised: (71) we used the fact that for all i ∈ [ p ] , either u i = 0 or z i = 0 , and in eq. (72) we used u , z ≥ 0 . Finally, since ˜ a = u (cid:107) w + (cid:107) − z [Equation]

Original: (67) we used u , z ≥ 0 .
Revised: w − [Equation] , we have

Original: Finally, in the notation of ?? ,
Revised: [Equation] which completes the proof. C R ELATIONSHIP TO NONLINEAR MAX - MARGIN Lemma C.1.

Revised: Let ( X , y ) be an orthogonally separable dataset, let w + , w − be deﬁned as in eqs.

Revised: (6) and (7) and let

Original: [Equation]
Revised: [Equation]

Original: where the positivity follows from a + ( t ) , ¯ u ≥{+i = 
 or z i =+} 0  .
Revised: for some u , z ∈ R p + such that u {+i = 
 or z i =+} 0 for all i ∈ [ p ] .

Revised: Also let u , z be normalised such that (cid:107) u (cid:107)

Revised: = (cid:107) w + (cid:107) − 1 / 2 and (cid:107)

Revised: z (cid:107) = (cid:107) w − (cid:107) − 1 / 2 .

Revised: Then ˜ θ (cid:44) (cid:8)(cid:102) W , ˜ a (cid:9) is a KKT point of the following constrained optimisation problem:

Revised: [Equation]

Revised: Proof.

Revised: By standard linear max-margin considerations (e.g. Hastie et al.

Revised: (2008, Section 4.5.2)), we know that

Revised: [Equation]

Original: {+for some α i ≥ 0 such that α i = 
+}
Revised: {+for some α i ≥ 0 such that α i = 
+}

Revised: if x i is a non-support vector.

Revised: It follows by orthogonal separability that for i with y i = 1 ,

Revised: [Equation]

Revised: and for i with y i = − 1 ,

Revised: [Equation]

Revised: [Equation]

Revised: we will need these properties shortly.

Revised: Let us now turn to checking the KKT status of ˜ θ wrt.

Revised: eq.

Revised: We start by showing that ˜ θ is feasible.

Revised: Let x i be such that y i = 1 ; then

Revised: [Equation]

Revised: where the last inequality follows from the deﬁnition of w + , eq.

Revised: Similarly, if x i is such that y i = − 1 , then

Revised: [Equation]

Original: By similar considerations we obtain that ˜  a (cid:124) z ≤ 0  , completing the proof.
Revised: This shows that ˜ θ is feasible. Next, we show that ˜ θ is a KKT point, i.e. we show that there exist λ 1 , . . . , λ n ≥ 0 such that for all i ∈ [ n ] , λ i (cid:0)

Revised: y

Revised: i f

Revised: ˜ θ ( x i ) − 1 (cid:1) = 0 , and 2.

Revised: ˜ θ ∈ (cid:80) ni

Original: {+=
+}
Revised: {+=
+}

Revised: λ i y i ∂ θ f ˜ θ ( x i ) ,

Revised: where ∂ θ f ˜ θ ( x ) denotes the Clarke subdifferential of f θ ( x ) wrt.

Revised: θ , evaluated at ˜ θ .

Revised: Speciﬁcally, we show that the choice

Revised: [Equation]

Revised: satisﬁes both conditions above.

Revised: As for the ﬁrst condition, observe that if i is a non-support example, then λ i

Revised: = α

Original: {+i = 
+}
Revised: {+i = 
+}

Revised: and the condition holds.

Revised: If i is a support example, then by eqs. (85) and (88), y i f

Revised: ˜ θ ( x i ) =

Revised: 1 and the condition holds as well.

Revised: As for the second condition, denote

Revised: [Equation]

Revised: where I θ ( x ) =

Revised: diag [ 1 { Wx i > 0 } ]

Revised: ∈ R p × p is the diagonal matrix whose ( i, i ) -element is one if w (cid:124) i x > 0 and zero otherwise.

Revised: It holds that g i (˜ θ ) ∈

Revised: ∂ θ f ˜ θ ( x i ) and n (cid:88)

Revised: [Equation]

Revised: This proves the second condition, and shows that ˜ θ is a KKT point of eq.

Revised: D E XPERIMENTS ON REAL DATA

Revised: In this section we explore the applicability of our result to real-world datasets and architectures (which lie outside the scope formally covered by our assumptions).

Revised: We experiment on the MNIST dataset subsetted to two classes, the digit 0 and the digit 1 .

Revised: We train a network consisting of six convolutional layers followed by two fully-connected layers.

Revised: We view the six convolutional layers as a ‘feature extractor’ and the two fully-connected layers as a two-layer fully-connected network of the kind we analyse in this paper.

Revised: The details of the architecture are given in Table 1.

Revised: We train the network by Adam with the binary cross-entropy loss and a batch size of 128.

Revised: We train for 50 epochs.

Revised: Prior to training, we multiply the weights of the fully-connected layers by 0.05, to approximate the small-norm initialisation assumed by theory.

Revised: [Table]

Revised: n, o) we denote a fully-connected layer whose input dimension is n and whose output dimension is o .

Revised: We conduct two sub-studies.

Revised: First, we demonstrate that the network learns orthogonally separable representations all by itself, in the course of training.

Revised: This is shown in Figure 3.

Revised: The ﬁrst subplot shows three distributions: The blue distribution is the distribution of x (cid:124) i x j where x i is sampled from class 0 and x j is sampled from class 1 .

Revised: The orange (or green) distribution is the distribution of x (cid:124) i x j where both x i , x j are sampled from class 0 (or class 1 ).

Revised: The other subplots show analogous inner-product distributions for the intermediate representations or learned features of the data, i.e. f l θ ( x i ) (cid:124)

Revised: f l θ ( x j ) instead of x (cid:124) i x j .

Revised: What we see is that the network learns representations such that examples of the same class are more similar to each other than examples of different classes – the orange and green distributions are generally more to the right compared to the blue distribution.

Revised: Moreover, higher-layer representations

Revised: [Figure]

Revised: are generally more strongly separated – as we move up the layer hierarchy, the orange and green distributions keep shifting rightward, whereas the blue distribution shifts leftward.

Revised: Remarkably, the 7th layer representations are orthogonally separable.

Revised: In the second sub-study, we explore properties of the weight matrix learnt by the ﬁrst linear layer of the network, in analogy to the ﬁrst-layer weight matrix in a two-layer net.

Revised: Figure 4a shows the top ten singular values of the weight matrix W 7 ∈ R 128 × 3136 .

Revised: We see that despite its size, it has very few (perhaps ﬁve or ten) signiﬁcantly non-zero singular values.

Revised: This is similar to what we observed for synthetic data in Section 5, though the separation between small and large singular values is less crisp and there are more than two non-zero values.

Revised: Figure 4b shows the rows (neurons) of

Revised: W 7 projected onto the top two singular dimensions (note that unlike in Section 5, the projection is lossy).

Revised: The neurons roughly form three clusters: a mixed cluster close to the origin and two clusters corresponding to positive and negative outer-layer weights.

Revised: Compared to our observations from Section 5, there is less variation in the neurons’ norms, leading to them forming clusters rather than rays.

Revised: This deviation from the theoretical prediction could be due to a number reasons, e.g. the use of biases, convolutional layers, or the large dimensionality of the layer.

Revised: We leave a detailed investigation of this question to future work.

Revised: [Figure]