I NDUCTIVE B IAS OF G RADIENT D ESCENT FOR E XPONENTIALLY W EIGHT N ORMALIZED S MOOTH H OMOGENEOUS N EURAL N ETS
Anonymous authors
Paper under double-blind review
A BSTRACT
We analyze the inductive bias of gradient descent for weight normalized smooth homogeneous neural nets, when trained on exponential or cross-entropy loss.
Our analysis focuses on exponential weight normalization (EWN), which encourages weight updates along the radial direction.
This paper shows that the gradient ﬂow path with EWN is equivalent to gradient ﬂow on standard networks with an adaptive learning rate, and hence causes the weights to be updated in a way that prefers asymptotic relative sparsity.
These results can be extended to hold for gradient descent via an appropriate adaptive learning rate.
The asymptotic convergence rate of the loss in this setting is given by Θ( 1 t (log t ) 2 ) and is independent of the depth of the network.
We contrast these results with the inductive bias of standard weight normalization (SWN) and unnormalized architectures, and demonstrate their implications on synthetic data sets.
Experimental results on simple data sets and architectures support our claim on sparse EWN solutions, even with SGD.
This demonstrates its potential applications in learning prunable neural networks.
I NTRODUCTION
The prevailing hypothesis for explaining the generalization ability of deep neural nets, despite their ability to ﬁt even random labels (Zhang et al., 2017), is that the optimisation/training algorithms such as gradient descent have a ‘bias’ towards ‘simple’ solutions.
This property is often called inductive bias, and has been an active research area over the past few years.
It has been shown that gradient descent does indeed seem to prefer ‘simpler’ solutions over more ‘complex’ solutions, where the notion of complexity is often problem/architecture speciﬁc.
The predominant line of work typically shows that gradient descent prefers a least norm solution in some variant of the L 2 -norm.
This is satisfying, as gradient descent over the parameters abides by the rules of L 2 geometry, i.e. the weight vector moves along direction of steepest descent, with length measured using the Euclidean norm.
However, there is nothing special about the Euclidean norm in the parameter space, and hence several other notions of ‘length’ and ‘steepness’ are equally valid.
In recent years, several alternative parameterizations of the weight vector, such as Batch normalization and Weight normalization, have seen immense success and these do not seem to respect L 2 geometry in the ‘weight space’.
We pose the question of inductive bias of gradient descent for some of these parameterizations, and demonstrate interesting inductive biases.
In particular, it can still be argued that gradient descent with these reparameterizations prefers simpler solutions, but the notion of complexity is different.
1. C ONTRIBUTIONS
The three main contributions of the paper are as follows.
• We establish that the gradient ﬂow path with exponential weight normalization is equal to the gradient ﬂow path of an unnormalized network using an adaptive neuron dependent learning rate.
This provides a crisp description of the difference between exponential weight normalized networks and unnormalized networks.
• We establish the inductive bias of gradient descent on standard weight normalized and exponentially weight normalized networks and show that exponential weight normalization is likely to lead to asymptotic sparsity in weights.
• We provide tight asymptotic convergence rates for exponentially weight normalized networks.
R ELATED W ORK 2. I NDUCTIVE B IAS
Soudry et al.
(2018) showed that gradient descent(GD) on the logistic loss with linearly separable data converges to the L 2 maximum margin solution for almost all datasets.
These results were extended to loss functions with super-polynomial tails in Nacson et al.
(2019b).
Nacson et al.
(2019c) extended these results to hold for stochastic gradient descent(SGD) and Gunasekar et al.
(2018a) extended the results for other optimization geometries.
Ji & Telgarsky (2019b) provided tight convergence bounds in terms of dataset size as well as training time.
Ji & Telgarsky (2019a) provide similar results when the data is not linearly separable.
Ji & Telgarsky (2019c) showed that for deep linear nets, under certain conditions on the initialization, for almost all linearly separable datasets, the network, in function space, converges to the maximum margin solution.
Gunasekar et al.
(2018b) established that for linear convolutional nets, under certain assumptions regarding convergence of gradients etc, the function converges to a KKT point of the maximum margin problem in fourier space.
Nacson et al.
(2019a) shows that for smooth homogeneous nets, the network converges to a KKT point of the maximum margin problem in parameter space.
Lyu & Li (2020) established these results with weaker assumptions and also provide asymptotic convergence rates for the loss.
Chizat & Bach (2020) explore the inductive bias for a 2-layer inﬁnitely wide ReLU neural net in function space and show that the function learnt is a max-margin classiﬁer for variation norm.
2. N ORMALIZATION
Salimans & Kingma (2016) introduced weight normalization and demonstrated that it replicates the convergence speedup of BatchNorm.
Similarly, other normalization techniques have been proposed as well(Ba et al., 2016)(Qiao et al., 2020)(Li et al., 2019), but only a few have been theoretically explored.
Santurkar et al.
(2018) demonstrated that batch normalization makes the loss surface smoother and L 2 normalization in batchnorm can even be replaced by L 1 and L ∞ normalizations.
Kohler et al.
(2019) showed that for GD, batchnorm speeds up convergence in the case of GLM by splitting the optimization problem into learning the direction and the norm.
Cai et al.
(2019) analyzed GD on BN for squared loss and showed that it converges for a wide range of lr.
Bjorck et al.
(2018) showed that the primary reason BN allows networks to achieve higher accuracy is by enabling higher learning rates.
Arora et al.
(2019) showed that in case of GD or SGD with batchnorm, lr for scale-invariant parameters does not affect the convergence rate towards stationary points.
Du et al.
(2018) showed that for GD over one-hidden-layer weight normalized CNN, with a constant probability over initialization, iterates converge to global minima.
Qiao et al.
compared different normalization techniques from the perspective of whether they lead to points, where neurons are consistently deactivated.
Wu et al.
(2019) established the inductive bias of gradient ﬂow with weight normalization for overparameterized least squares and showed that for a wider range of initializations as compared to normal parameterization, it converges to the minimum L 2 norm solution.
Dukler et al.
(2020) analyzed weight normalization for multilayer ReLU net in the inﬁnite width regime and showed that it may speedup convergence.
Some other papers(Luo et al., 2019;
Roburin et al., 2020) also provide other perspectives to think about normalization techniques.
P ROBLEM S ETUP
We use a standard view of neural networks as a collection of nodes/neurons grouped by layers.
Each node u is associated with a weight vector w u , that represents the incoming weight vector for that node.
In case of CNNs, weights can be shared across different nodes.
w represents all
[Figure] [Figure]
(b) SWN 0.
[Figure]
the parameters of the network arranged in form of a vector.
The dataset is represented in terms of ( x i , y i ) pairs and m represents the number of points in the dataset.
The function represented by the neural network is denoted by Φ( w , . ) .
The loss for a single data point x i is given by (cid:96) ( y i , Φ( w , x i )) and the loss vector is represented by (cid:96) .
The overall loss is represented by L ( w ) and is given by
L ( w ) =
(cid:80) mi
=1 (cid:96) ( y i , Φ( w , x i )) .
We sometimes abbreviate L ( w ( t )) as L when the context is clear.
In standard weight normalisation (SWN), each weight vector w u is reparameterized as γ u v u (cid:107) v u (cid:107) .
This was proposed by Salimans & Kingma (2016), as a substitute for Batch Normalization and has been practically used in multiple papers such as Sokolic et al.
(2017), Dauphin et al.
(2017), Kim et al.
(2018) and Hieber et al.
The corresponding update equations for gradient descent are given by
[Equation]
In exponential weight normalisation (EWN), each weight vector w u is reparameterized as e α u v u (cid:107) v u (cid:107) .
This was mentioned in Salimans & Kingma (2016), but to the best of our knowledge, has not been widely used.
The corresponding update equations for gradient descent with learning rate η ( t ) are given by
[Equation]
The update equations for gradient ﬂow are the continuous counterparts for the same.
In case of gradient ﬂow, for both SWN and EWN, we assume (cid:107) v u (0) (cid:107) = 1 , to simplify the update equations.
I NDUCTIVE B IAS OF W EIGHT N ORMALIZATION
In this section, we state our main results for weight normalized smooth homogeneous models on exponential loss( (cid:96) ( y i , Φ( w , x i ) =
e − y i Φ( w , x i ) ).
The results for cross-entropy loss and proofs have been deferred to the appendix due to space constraints.
First, we state the main proposition that helps in establishing these results for EWN.
Theorem 1.
The gradient ﬂow path with learning rate η ( t ) for EWN and SWN are given as follows:
[Equation]
Thus, the gradient ﬂow path of EWN can be replicated by an adaptive learning rate given by η ( t ) (cid:107) w u ( t ) (cid:107) 2 on unnormalized network(Unnorm).
These parameterizations also induce different neighborhoods in the parameter space, that have been shown in Figure 1.
A SSUMPTIONS
The assumptions in the paper can be broadly divided into loss function/architecture based assumptions and trajectory based assumptions.
The loss functions/architecture based assumptions are shared across both gradient ﬂow and gradient descent.
Loss function/Architecture based assumptions
[Equation]
) , for some
[Equation]
and L >
[Equation]
Gradient ﬂow .
For gradient ﬂow, we make the following trajectory based assumptions
[Equation]
A1)
[Equation]
ρ >
[Equation]
The ﬁrst assumption is typically satisﬁed in scenarios where a positively homogeneous network achieves 100% training accuracy.
This is not a completely unreasonable assumption, given recent papers demonstrating neural networks with sufﬁcient overparameterization can ﬁt even random labels(Zhang et al.
(2017), Jacot et al.
(2018)), and is a standard assumption made when the purpose is to ﬁnd the inductive bias.
The second assumption states that the network converges in direction and this has been recently shown in Ji & Telgarsky (2020) to hold for gradient ﬂow on homogeneous neural nets without normalization under some regularity assumptions.
The third and fourth assumptions are required to show convergence of the gradients in direction.
The fourth assumption is indeed true for SWN as shown in Lyu & Li (2020) 1 .
Gradient Descent .
For gradient descent, we also require the learning rate η ( t ) to not grow too fast.
[Equation]
Proposition 1.
Under assumptions (A1)-(A4), lim t →∞ η ( t ) (cid:107) w u ( t ) (cid:107)∇ w u L ( w ( t )) (cid:107) =
holds for every u in the network with η ( t )
=
O ( 1 L c ) , where c < 1 .
This proposition establishes that the assumption (A5) is mild and holds for constant η ( t ) , that is generally used in practice.
While some of these assumptions are non-standard we believe they do generally hold, and demonstrate the viability of these assumptions in a toy experiment which we call Lin-Sep .
In this experiment a 2-layered EWN neural network, with 8 neurons in the hidden layer and a ReLU-squared activation function, is trained on a linearly separable dataset.
The learning rate schedule used was
O ( 1 L 0 . 97 ) and the network was trained till a loss of e − 300 .
The corresponding graphs for EWN are shown in Figure 2.
Similar results for SWN have been deferred to Figure 8 in the appendix.
4. E FFECT OF N ORMALISATION ON W EIGHT AND G RADIENT N ORMS
This section contains the main theorems and the difference between EWN and SWN that makes
EWN asymptotically relatively sparse as compared to SWN.
First, we will state a common proposition for both SWN and EWN.
Proposition 2.
Under assumptions (A1)-(A4) for gradient ﬂow and (A1)-(A5) for gradient descent, for both SWN and EWN, the following hold:
[Equation]
[Figure]
it can be seen that only weights
8 keep on growing in
So only they keep evolving
Eventually
(ii) Let (cid:101) w u =
lim t →∞ w u ( t ) (cid:107) w ( t ) (cid:107) and (cid:101) g
u
=
lim t →∞ −∇ w u L ( w ( t ))
(cid:107)∇ w L ( w ( t )) (cid:107) .
Then, (cid:101) w u = λ (cid:101) g u for some
λ ≥
The ﬁrst and second part state that under the given assumptions, for both SWN and EWN, gradients converge in direction and the weights that contribute to the ﬁnal direction of w , converge in opposite direction of the gradients.
Now, we provide the main theorem that distinguishes SWN and EWN.
Theorem 2.
Under assumptions (A1)-(A4) for gradient ﬂow and (A1)-(A5) for gradient descent the following hold
(i) for EWN, (cid:107) (cid:101) w u (cid:107) > 0 , (cid:107) (cid:101) w v (cid:107) > 0 =
⇒ lim
t →∞ (cid:107) w u ( t ) (cid:107)(cid:107)∇ w u L ( w ( t )) (cid:107) (cid:107) w v ( t ) (cid:107)(cid:107)∇ w v L ( w (
[Table]
)) (cid:107) = (cid:107) (cid:101) w u
[Table]
(cid:101) g u (cid:107) (cid:107) (cid:101) w v (cid:107)(cid:107) (cid:101) g v (cid:107) =(ii) for SWN, (cid:107) (cid:101) w u (cid:107) > 0 , (cid:107) (cid:101) w v (cid:107) > 0 = ⇒
lim t →∞ (cid:107) w u ( t ) (cid:107)(cid:107)∇ w v L ( w ( t )) (cid:107) (cid:107) w v ( t ) (cid:107)(cid:107)∇ w u L ( w ( t )) (cid:107) =
(cid:107) (cid:101)
w
[Table]
g
[Table]
w
[Table]
=
Thus, asymptotically, for EWN, (cid:107) w u ( t ) (cid:107) =
k 1 ( t ) (cid:107)∇ w u L ( w ( t ))
(cid:107) while for SWN, (cid:107) w u ( t ) (cid:107) =
k 2 ( t ) (cid:107)∇ w u L ( w ( t ))
(cid:107) , where k 1 ( t ) and k 2 ( t ) are independent of the neuron u .
We demonstrate this property of EWN on the Lin-Sep experiment in Figure 3.
The results for SWN have been deferred to Figure 9 in the Appendix.
Now, we provide a corollary for the case of multilayer linear nets.
Corollary 1.
Consider a weight normalized(SWN or EWN) multilayer linear net, represented by y = W n W n − 1 ...
W 1 x .
Under assumptions (A1)-(A4) for gradient ﬂow and (A1)-(A5) for gradient descent, if the dataset is sampled from a continuous distribution w.r.t R d , then, with probability 1, θ = W (cid:62) 1 W (cid:62) 2 ....W (cid:62) n converges in direction to the maximum margin separator for all linearly separable datasets.
[Figure] [Figure]
C
o s i n e b / w w u a n d g (a)
[Figure]
C
o s i n e b / w w u a n d w u w 5 ww(b) 1.
[Figure]
.0 log ||
w u
|| 190.
l o g
||
w u
|| x
[Figure] [Figure]
w
[Figure]
(c)
[Figure] [Figure]
[Figure]
log || w u ||
[Figure]
[Figure]
l o g ||
w u
||
x
+ y
[Figure] [Figure]
w 5 w 7 w(d) 1.
2.2 log ||
w u
|| 286.
l o g
||
w u
||
x
+ y = -284.
w 5 w 7 w(e)
+ log (cid:107)∇ w u L(cid:107) is approximately same.
4. S PARSITY I NDUCTIVE B IAS FOR E XPONENTIAL W EIGHT N ORMALISATION
The inverse relation between (cid:107) w u ( t ) (cid:107) and (cid:107)∇ w u L ( w ( t )) (cid:107) in the EWN trajectory results in an interesting inductive bias that favours movement along sparse directions.
Proposition 3.
Let assumptions (A1)-(A5) be satisﬁed.
Consider two nodes u and v in the network such that (cid:107) (cid:101)
g v (cid:107)
≥ (cid:107) (cid:101) g u (cid:107) > 0 and (cid:107) w u ( t ) (cid:107) , (cid:107) w v ( t ) (cid:107) → ∞ .
Let (cid:107) (cid:101)
g u (cid:107) (cid:107) (cid:101) g v (cid:107) be denoted by c .
Let (cid:15), δ be such that 0 < (cid:15) <
c and 0 < δ < 2 π .
Then, the following holds:
There exists a time t 1 , such that for all t > t 1 both SWN and EWN trajectories have the following properties: (a) (cid:107)∇ w u L ( w ( t ))
(cid:107) (cid:107)∇ w v L ( w ( t ))
(cid:107) ∈ [ c − (cid:15), c
+ (cid:15) ] (b) (cid:16) w u ( t ) (cid:107) w u ( t ) (cid:107) (cid:17) (cid:62) (cid:16) −∇ w u L ( w ( t ))
(cid:107)∇ w u L ( w ( t )) (cid:107) (cid:17) ≥ cos( δ ) (c) (cid:16) w v ( t ) (cid:107) w v ( t ) (cid:107) (cid:17) (cid:62)
[Table]
for SWN, lim t →∞
[Table]
=
c 3.
for EWN, if at some time t 2 > t 1 , (a) (cid:107) w u ( t 2 ) (cid:107) (cid:107) w v ( t 2 ) (cid:107) >
[Equation]
[Equation] (b)
[Table]
w u
[Table] [Table]
[Table]
[Equation] [Table]
δ
[Table]
+ [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Figure] [Figure]
The above proposition shows that the limit property of the weights in Theorem 2, makes non-sparse w an unstable convergent direction for EWN.
But that is not the case for SWN.
We demonstrate the relative sparsity between EWN, SWN and Unnorm through two toy experiments – Simple-Traj and XOR .
In the Simple-Traj experiment, we have a single data point at (2 , 1) , that is labelled positive and train a network with linear activations.
The architecture is shown in Figure 4a, where weights in blue and red are frozen to values 1 and 0 respectively.
Thus, there are effectively only two scalar parameters- w 1 and w 2 .
The network is trained till a loss value of e − 50 starting from 5 different initialization points.
The weight trajectories in Figure 4b shows that EWN prefers to converge either along the x or y axis, and hence has an asymptotic relative sparsity property.
In the XOR experiment, we train a 2-layer ReLU network, with 20 hidden neurons on XOR dataset(shown in Figure 5a).
The second layer is ﬁxed to the values 1 or -1 randomly.
For attaining 100% accuracy on this dataset with this architecture, at least 4 hidden units are needed.
As can be seen in Figure 5, EWN asymptotically uses exactly 4 neurons out of 20, while Unnorm uses all the 20 neurons.
The results for SWN have been deferred to Figure 10 in the appendix.
C ONVERGENCE R ATES
Under Assumption (A2), w can be represented as w = g ( t ) (cid:101) w + r ( t ) , where lim t →∞ (cid:107) r ( t ) (cid:107) g ( t ) = 0 .
Let d : N → R , given by d ( t ) =
(cid:80) tτ
=
η ( τ ) denote total step size.
[Figure]
The asymptotic convergence rate of loss for SWN and Unnorm have already been established in Lyu & Li (2020) as Θ
[Equation]
For EWN, the corresponding theorem is provided below
Theorem 3.
For EWN, under Assumptions (A1)-(A5) and lim t →∞ (cid:107) r ( t
[Equation]
1) − r
[Equation]
t )
[Equation]
g
[Equation]
t
[Equation]
− g ( t ) = 0 , the following hold
(cid:107) w ( t ) (cid:107) asymptotically grows at Θ (cid:16) (log( d ( t ))
1 L
[Equation]
L ( w ( t )) asymptotically goes down at the rate of Θ (cid:16)
[Equation]
For multilayer linear nets, the variation of convergence rate with number of layers for a linearly separable dataset is illustrated in Figure 6.
All of these networks were explicitly initialized to represent the same point in function space.
It can be seen that EWN, SWN and unnormalized networks all converge faster with more layers, but the effect is much less pronounced for EWN.
MNIST P RUNING E XPERIMENTS
As EWN leads to asymptotically sparse solutions, it is likely that a sufﬁciently trained EWN network would be comparatively robust to pruning.
In this section, we compare the pruning efﬁcacy of EWN, SWN and Unnorm on a 2-layer ReLU network trained on the MNIST dataset.
In case of
EWN and SWN, only the ﬁrst layer is weight normalized as only this layer needs to be pruned.
The pruning criterion used is the difference between the initial and ﬁnal weight norm, i.e, the weights that grow the least in norm are pruned ﬁrst.
The corresponding pruning graphs at different loss values are shown in Figure 7.
It can be seen that when the loss levels are sufﬁciently low, the EWN network becomes better adapted for pruning, signiﬁcantly outperforming SWN and the unnormalized network in terms of test accuracy for a given level of pruning.
The variation of norm of the weight vectors with gradient descent steps for neurons in the ﬁrst layer has been deferred to Figure 11 in the appendix.
C ONCLUSION
In this paper, we analyze the inductive bias of weight normalization for smooth homogeneous neural nets and show that exponential weight normalization is likely to lead to asymptotically sparse solutions and has a faster convergence rate than unnormalized or standard weight normalized networks.
. Understanding batch normalization.
In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi and R. Garnett (eds.),
Information Processing Systems 31 , pp.
Curran Associates, Inc.,
A P ROOF OF T HEOREM
Theorem.
The gradient ﬂow path with learning rate η ( t ) for EWN and SWN are given as follows:
[Equation]
The proof for the two parts will be provided in different subsections, where the corresponding part will be restated for ease of the reader.
A. E XPONENTIAL W EIGHT N ORMALIZATION
Theorem.
The gradient ﬂow path with learning rate η ( t ) for EWN is given by:
[Equation]
Proof.
In case of EWN, weights are reparameterized as w u =
e
α u v u (cid:107) v u (cid:107) .
Then
[Equation]
Now, in case of gradient ﬂow with learning rate η ( t ) , we can say
[Equation]
Now, using these equations, we can say
[Equation]
Thus, (cid:107) v u ( t ) (cid:107) does not change with time.
As we assumed (cid:107) v u (0) (cid:107) to be 1, therefore for any t, (cid:107) v u
[Equation]
t )
[Equation]
=
[Equation]
Using this simpliﬁcation, we can write
[Equation]
Thus
[Equation]
the gradient ﬂow path with exponential weight normalization can be replicated by an adaptive learning rate given by η ( t ) (cid:107) w u ( t ) (cid:107) 2 .
A. S TANDARD W EIGHT N ORMALIZATION
Theorem.
The gradient ﬂow path with learning rate η ( t ) for SWN is given by:
[Equation]
Proof.
In case of SWN
[Equation]
weights are reparameterized as w u = γ u v u (cid:107) v u (cid:107) .
Then
[Equation]
Now, in case of gradient ﬂow with learning rate η ( t )
[Equation]
we can say
[Equation]
Now, similar to EWN, (cid:107) v u ( t ) (cid:107) does not change with time.
Using the fact that (cid:107) v u ( t ) (cid:107) = 1 for all t , we can say
[Equation]
Replacing γ u ( t ) by (cid:107) w u ( t ) (cid:107) gives the required expression.
B P ROOF OF P ROPOSITION
Proposition.
Under assumptions (A1)-(A4), lim t →∞ η ( t ) (cid:107) w u ( t ) (cid:107)∇ w u L ( w ( t )) (cid:107) =
holds for every u in the network with η ( t )
=
O ( 1 L c ) , where c < 1 .
Proof.
Under assumption (A1) and (A2), w can be represented as w = g ( t ) (cid:101) w + r ( t ) , where
[Equation]
r ( t ) (cid:107) g
( t )
= 0 .
Now, for exponential loss,
[Equation]
From assumption (A4), we know Φ( (cid:101) w , x i ) ≥ ρ for all i .
Now, using Euler’s homogeneous theorem, we can say
[Equation]
Thus, (cid:107)∇ w Φ( (cid:101) w , x i ) (cid:107) > 0 for all i .
Now, using the equations above and assumption (A3), we can say
[Equation]
where k 1 is some constant.
Now, if η ( t ) =
O ( 1 L ( w ( t )) c ) , where c < 1 , then using the fact that L goes down at the rate of e − ρg ( t ) L and (cid:107) w (cid:107) goes up at the rate of g ( t ) , we can say
[Equation]
t
[Equation]
k 2 e − ρg ( t ) L g ( t ) L − 1 (cid:107) (cid:80) mi =1 (cid:101) (cid:96)
i y
i ∇ w Φ( (cid:101) w , x i ) (cid:107)(cid:107) w ( t ) (cid:107)
L ( w ( t ))
c =
C P ROOF OF P ROPOSITION
Proposition.
Under assumptions (A1)-(A4) for gradient ﬂow and (A1)-(A5) for gradient descent, for both SWN and EWN, the following hold:
(i)
lim
t →∞ −∇ w L ( w ( t ))
(cid:107)∇ w L ( w ( t ))
(cid:107) =
µ (cid:80) mi =1 (cid:101) (cid:96) i y i ∇ w Φ( (cid:101) w , x i ) =
(cid:101) g , where µ > 0 .
(ii) Let (cid:101) w u =
lim t →∞ w u ( t ) (cid:107) w ( t ) (cid:107) and (cid:101) g
u
=
lim t →∞ −∇ w u L ( w ( t ))
(cid:107)∇ w L ( w ( t ))
(cid:107)
Then, (cid:101) w u = λ (cid:101) g u for some λ ≥
The proof for different cases will be split into different subsections and corresponding proposition will be stated there for ease of the reader.
The proof will depend on the Stolz Cesaro theorem(stated in Appendix J),Integral Form of Stolz-Cesaro Theorem(stated and proved in Appendix J) and following lemmas that have been proved in Appendix I.
Lemma 1.
Under assumptions (A1)-(A4) for gradient ﬂow and (A1)-(A5) for gradient descent, for both SWN and EWN, (cid:101) w (cid:62) u (cid:101) g u ≥ 0 for all nodes u in the network.
Lemma 2.
Under assumptions (A1)-(A4) for gradient ﬂow and (A1)-(A5) for gradient descent, for both SWN and EWN, there exists atleast one node u in the network satisfying (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) g u (cid:107) > 0 .
Lemma 3.
Consider two unit vectors a and b satisfying a (cid:62) b ≥ 0 and a (cid:62) b < 1 .
Then, there exists a small enough (cid:15) > 0 , such that for any unit vector c satisfying c (cid:62) a ≥ cos( (cid:15) ) and any unit vector d satisfying d (cid:62) b ≥ cos( (cid:15) ) , b (cid:62) ( I − cc (cid:62) )
d ≥ (cid:15) .
Lemma 4.
Consider sequence a satisfying the following properties
a k >2.
(cid:80) ∞ k =
a k = ∞ 3.
lim k →∞ a k =
Then (cid:80) ∞
k =
a k (cid:113)(cid:80) kj =
a 2 j = ∞
Lemma 5.
Consider two sequences a and b satisfying the following properties
a k > 0 , (cid:80) ∞
k =0 a k = ∞ and lim k →∞ a k
=2.
b
> 0 , b is increasing and b 2 k +1 ≤ b 2 k
+ ( a k b k )
Then (cid:80) ∞
k =
a k b k = ∞ .
Lemma 6.
Consider two sequences a and b satisfying the following properties
a k > 0 and (cid:80) ∞
k =0 a k = ∞ 2.
b k > 0 and (cid:80) ∞
k =
b k = ∞ 3.
(cid:80) ∞ k =
( a k − b k ) converges to a ﬁnite value 4.
lim k →∞ a k b k exists
Then lim k →∞ a k b k = 1 .
C. E XPONENTIAL W EIGHT N ORMALIZATION
In this section, we will use e α u ( t ) and (cid:107) w u ( t ) (cid:107) interchangeably.
C.1. G RADIENT F LOW
Proposition.
Under assumptions (A1)-(A4) for gradient ﬂow, for EWN, the following hold:
(i)
lim
t →∞ −∇ w L ( w ( t ))
(cid:107)∇ w L ( w ( t ))
(cid:107) =
µ (cid:80) mi =1 (cid:101) (cid:96) i y i ∇ w Φ( (cid:101) w , x i ) =
(cid:101) g , where µ > 0 .
(ii) Let (cid:101) w u =
lim t →∞ w u ( t ) (cid:107) w ( t ) (cid:107) and (cid:101) g
u
=
lim t →∞ −∇ w u L ( w ( t ))
(cid:107)∇ w L ( w ( t )) (cid:107) .
Then, (cid:101) w u = λ (cid:101) g u for some λ ≥
Update Equations:
[Equation] [Equation]
Proof.
Using the fact that ρ > 0 and Euler’s homogeneous theorem, we can say
[Equation]
Thus, (cid:107)∇ w
[Equation]
for all i .
Let w ( t ) =
g ( t ) (cid:101) w + r ( t ) , where lim t →∞ (cid:107) r ( t ) (cid:107) g ( t ) = 0 .
Now, by Taylor’s Theorem, we can say
[Equation]
Now, ∇ 2 Φ( (cid:101) w + k r ( t ) g
( t ) , x i ) can be bounded by a constant and lim t
[Equation]
= 0 .
Thus, lim
[Equation]
Φ( (cid:101) w + k r ( t ) g ( t ) , x i ) r
( t ) g ( t ) dk
= 0 .
Thus, we can say, if (cid:107)∇ w Φ( (cid:101) w , x i ) (cid:107) > 0 for some i , then
[Equation]
Now,
[Equation]
Now, Let S
= { i : y i Φ( (cid:101) w , x i ) =
min
j y j Φ( (cid:101) w , x j ) } .
Let (cid:15) denote min j/ ∈ S
y j Φ( (cid:101) w , x j ) − ρ .
Consider a ∈ S and b / ∈ S , then
[Equation]
Now, as the minimum difference is (cid:15) , therefore lim t →∞ (cid:96) b ( w
[Equation]
t
[Equation]
) (cid:96) a ( w ( t ) = 0 .
Thus, ∀ j / ∈ S, (cid:101) (cid:96) j = 0 .
Now, using Equation (9) and the expression for −∇ w L ( w ( t )) from before, we can say
[Equation]
µ (cid:88) i ∈ S (cid:101) (cid:96)
i y
i ∇ w Φ( (cid:101) w , x i ) where µ = 1 (cid:107) (cid:80) i ∈ S (cid:101) (cid:96) i
( y i Φ( (cid:101) w ,x i )) (cid:107) .
( ii ) (cid:107)
(cid:101)
w u (cid:107) > 0 = ⇒ (cid:101) w u = λ (cid:101) g u for some λ >
Proof.
Consider a node u having (cid:107) (cid:101) w u (cid:107) > 0 .
The proof will be split into two parts depending on (cid:107) (cid:101) g u (cid:107) > 0 or (cid:107) (cid:101) g u (cid:107) = 0 .
Case 1: (cid:107) (cid:101) g u (cid:107) >
Let the angle between (cid:101) w u and (cid:101)
g u be denoted by ∆ .
Using Lemma 1, we can say ∆ ≤ π 2 .
We will prove the statement by contradiction, so let’s assume ∆ > 0 .
Now, we know, − ∇ w u L ( w ( t ))
(cid:107)∇ w u L ( w ( t )) (cid:107) converges to (cid:101) g u and v u ( t ) converges in direction of (cid:101) w u .
Taking dot product with (cid:101) g u (cid:107) (cid:101) g u (cid:107) on both sides of Equation (8) and using Lemma 3, we can say there exists a time t 1 and a small enough (cid:15) , such that for any t > t 1 ,
[Equation]
(cid:107) (cid:15) (10)
Now, using the fact that α u → ∞ and Equation (7), we can say
[Equation]
) (cid:107)∇
[Equation]
L
[Equation]
w ( t ))
(cid:107)
[Equation]
∞
Integrating Equation (10) on both the sides from t 1 to ∞ , we get
[Equation]
This is not possible as vectors on LHS have bounded norm.
This contradicts.
Hence ∆ = 0 .
[Equation]
We are going to show that it is not possible to have (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) g u (cid:107) = 0 .
Using Lemma 2, we can say there exists atleast one node v satisfying (cid:107) (cid:101) w v (cid:107) > 0 and (cid:107) (cid:101) g v (cid:107) > 0 .
Now, using Equation (7), we can say
[Equation]
From Case 1 , we know, for any (cid:15) > 0 , that there exists a time t 1 , such that for t > t 1 , (cid:16)
[Equation] [Equation]
≥ cos( (cid:15) ) .
Now, using Equation (7), we can say
[Equation]
Thus, we can say, for t > t 1 ,
[Equation]
t
[Equation]
Now, as (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) w v (cid:107) > 0 , therefore both the integrals diverge.
Also, the integrands converge in ratio to 0 as (cid:107) (cid:101) g u (cid:107) = 0 and (cid:107) (cid:101) g v (cid:107) > 0 .
Thus, taking limit t → ∞ on both the sides and using the Integral form of Stolz-Cesaro theorem, we can say
[Equation]
However, this is not possible as (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) w v (cid:107) > 0 .
This contradicts.
Therefore, such a case is not possible.
C.1. G RADIENT D ESCENT
Proposition.
Under assumptions (A1)-(A5) for gradient descent, for EWN, the following hold:
[Equation]
i
[Equation]
, where µ > 0 .
(ii) Let (cid:101) w u =
lim t →∞ w u ( t ) (cid:107) w ( t ) (cid:107) and (cid:101) g
u
=
lim t →∞ −∇ w u L ( w ( t ))
(cid:107)∇ w L ( w (
t ))
(cid:107)
Then,
(cid:101)
w u = λ (cid:101) g u for some λ ≥
Update Equations:
[Equation] [Equation]
y i ∇ w Φ( (cid:101) w , x i ) =
(cid:101) g , where µ >
Proof.
Follows exactly as shown for gradient ﬂow in Appendix C.1.1.
( ii ) (cid:107) (cid:101) w u (cid:107) > 0 =
⇒ (cid:101) w u = λ (cid:101) g u for some λ >
Proof.
Consider a node u having (cid:107) (cid:101) w u (cid:107) > 0 .
The proof will be split into two parts depending on (cid:107) (cid:101) g u (cid:107) > 0 or (cid:107) (cid:101) g u (cid:107) = 0 .
Case 1: (cid:107) (cid:101) g u (cid:107) > 0 .
Let the angle between (cid:101) w u and (cid:101)
g u be denoted by ∆ .
Using Lemma 1, we can say ∆ ≤ π 2 .
We will prove the statement by contradiction, so let’s assume ∆ > 0 .
Now, we know, − ∇ w u L ( w ( t ))
(cid:107)∇ w u L ( w ( t )) (cid:107) converges to (cid:101) g u and v u ( t ) converges in direction of (cid:101) w u .
Taking dot product with (cid:101) g u (cid:107) (cid:101) g u (cid:107) on both sides of Equation (12) and using Lemma 3, we can say there exists a time t 1 and a small enough (cid:15) , such that for any t > t 1 ,
[Equation]
[Equation]
However, in this case, (cid:107) v u ( t ) (cid:107) doesn’t stay constant and thus increase in dot product doesn’t directly correspond to an increase in angle.
Now, using Equation (12), we can say
[Equation]
Using the above two equations
[Equation]
we can say
[Equation]
for time t > t 1 ,
[Equation]
Unrolling the equation above
[Equation]
we get
[Equation]
Now, as α u ( t ) → ∞ , therefore, using Equation (11), we can say
[Equation]
Now, using this identity, along with the Assumption (A5), Equation (13) and Lemma 5, we can say
[Equation]
Using this along with Equation (14) and Lemma 4, we can say
[Equation]
However, this is not possible as the vectors on LHS have bounded norm.
This contradicts.
Thus ∆ = 0 .
Case 2: (cid:107) (cid:101) g u (cid:107) =
We are going to show that its not possible to have (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) g u (cid:107) = 0 .
By Lemma 3, we know there exists atleast one node s satisfying (cid:107) (cid:101) w s (cid:107) > 0 and (cid:107) (cid:101) g s (cid:107) > 0 .
Now from Equation (11), we can say
[Equation]
Thus,
[Equation]
Now, we know, α u ( t ) and α s ( t ) → ∞ .
Also, as (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) w s (cid:107) > 0 , therefore α u − α s converges.
Therefore the RHS of Equation (15) converges as well.
However, RHS is the difference of two diverging series.
Also, as v s ( t ) and ∇ w s L ( w ( t )) eventually get aligned and
[Equation]
= 0 , so, we can say
[Equation]
However, this contradicts Lemma 6, as the ratio must be converging to 1 if the limit exists.
Therefore this case is not possible.
C.
S TANDARD W EIGHT N ORMALIZATION C.2. G RADIENT FLOW
Proposition.
Under assumptions (A1)-(A4) for gradient ﬂow, for SWN, the following hold:
(i)
lim
t →∞ −∇
[Equation]
( w ( t )
[Equation]
(cid:107)∇
[Equation]
( w ( t )
[Equation]
(cid:107)
[Equation]
µ (cid:80) mi
[Equation]
1 (cid:101) (cid:96)
[Equation]
y
[Equation]
∇ w Φ( (cid:101) w , x i )
= (cid:101) g , where µ > 0 .
(ii) Let (cid:101) w u =
lim t →∞ w u ( t ) (cid:107) w ( t ) (cid:107) and (cid:101) g
u
=
lim t →∞ −∇ w u L ( w ( t ))
(cid:107)∇ w L ( w ( t )) (cid:107) .
Then, (cid:101) w u = λ (cid:101) g u for some
λ ≥
Update Equations:
[Equation] [Equation]
g
[Equation]
where µ >
Proof.
Follows exactly as shown for gradient ﬂow in Appendix C.1.1.
( ii ) (cid:107) (cid:101) w u (cid:107) > 0 =
⇒ (cid:101) w u = λ (cid:101) g u for some λ >
Proof.
Consider a node u having (cid:107) (cid:101) w u (cid:107) > 0 .
In this case, γ u ( t ) can either tend to ∞ or −∞ .
We will consider the case γ u ( t ) → ∞ .
The other case can be handled similarly.
The proof will be split into two parts depending on (cid:107) (cid:101) g u (cid:107) > 0 or (cid:107) (cid:101) g u (cid:107) = 0 .
Case 1: (cid:107)
[Equation]
>
Let the angle between (cid:101) w u and (cid:101)
g u be denoted by ∆ .
Using Lemma 1, we can say ∆ ≤ π 2 .
We will prove the statement by contradiction, so let’s assume ∆ > 0 .
Now, we know, − ∇ w u L ( w ( t ))
(cid:107)∇ w u L ( w ( t )) (cid:107) converges to (cid:101) g u and v u ( t ) converges in direction of (cid:101) w u .
Taking dot product with (cid:101) g u (cid:107) (cid:101) g u (cid:107) on both sides of Equation (17) and using Lemma 3, we can say there exists a time t 1 and a small enough (cid:15) , such that for any t > t 1 ,
[Equation]
Now, using the fact that γ u → ∞ and Equation (16), we can say
[Equation]
∞
[Equation]
t
[Equation]
L ( w ( t )
[Equation]
dt = ∞
Integrating Equation (18) on both the sides from t 1 to ∞ , we get (cid:18)
[Equation]
(cid:107)
[Equation]
≥ ∞
This is not possible as vectors on LHS have bounded norm.
This contradicts.
Hence ∆ = 0 .
[Equation]
[Equation]
We are going to show that it is not possible to have (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) g u (cid:107) = 0 .
Using Lemma 2, we can say there exists atleast one node s satisfying (cid:107) (cid:101) w s (cid:107) > 0 and (cid:107) (cid:101) g s (cid:107) > 0 .
Now, using Equation (16), we can say
[Equation]
From Case 1 , we know, for any (cid:15) > 0 , that there exists a time t 1 , such that for t > t 1 , (cid:16)
[Equation]
Now, using Equation (16), we can say
[Equation]
Thus, we can say, for t > t 1 ,
[Equation]
Now, as (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) w s (cid:107) > 0 , therefore both the integrals diverge.
Also, the integrands converge in ratio to 0 as (cid:107) (cid:101) g u (cid:107) = 0 and (cid:107) (cid:101) g v (cid:107) > 0 .
Thus, taking limit t → ∞ on both the sides and using the Integral form of Stolz-Cesaro theorem, we can say
[Equation]
However, this is not possible as (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) w s (cid:107) > 0 .
This contradicts.
Therefore, such a case is not possible.
C.2. G RADIENT D ESCENT
Proposition.
Under assumptions (A1)-(A5) for gradient descent, for SWN, the following hold
: (i) lim
t →∞ −∇ w L ( w ( t ))
(cid:107)∇ w L ( w ( t ))
(cid:107) =
µ (cid:80) mi =1 (cid:101) (cid:96) i y i ∇ w Φ( (cid:101) w , x i ) =
(cid:101) g , where µ > 0 .
(ii) Let (cid:101) w u =
lim t →∞ w u ( t ) (cid:107) w ( t ) (cid:107) and (cid:101) g
u
=
lim t →∞ −∇ w u L ( w ( t ))
(cid:107)∇ w L ( w ( t )) (cid:107) .
Then, (cid:101) w u = λ (cid:101) g u for some λ ≥
Update Equations:
[Equation]
( i ) lim
t
[Equation]
(cid:101) w
[Equation]
x i ) =
(cid:101) g , where µ >
Proof.
Follows exactly as shown for gradient ﬂow in Appendix C.1.
[Equation]
( ii ) (cid:107) (cid:101) w u (cid:107) > 0 =
⇒ (cid:101) w u = λ (cid:101) g u for some λ >
Proof.
Consider a node u having (cid:107) (cid:101) w u (cid:107) > 0 .
In this case, γ u ( t ) can either tend to ∞ or −∞ .
We will consider the case γ u ( t ) → ∞ .
The other case can be handled similarly.
The proof will be split into two parts depending on (cid:107) (cid:101) g u (cid:107) > 0 or (cid:107) (cid:101) g u (cid:107) = 0 .
Case 1: (cid:107) (cid:101) g u (cid:107)
>
Let the angle between (cid:101) w u and (cid:101)
g u be denoted by ∆ .
Using Lemma 1, we can say ∆ ≤ π 2 .
We will prove the statement by contradiction, so let’s assume ∆ > 0 .
Now, we know, − ∇ w u L ( w ( t ))
(cid:107)∇ w u L ( w ( t )) (cid:107) converges to (cid:101) g u and v u ( t ) converges in direction of (cid:101) w u .
Taking dot product with (cid:101) g u (cid:107) (cid:101) g u (cid:107) on both sides of Equation (20) and using Lemma 3, we can say there exists a time t 1 and a small enough (cid:15) , such that for any t > t 1 ,
[Equation]
However, in this case, (cid:107) v u ( t ) (cid:107) doesn’t stay constant and thus increase in dot product doesn’t directly correspond to an increase in angle.
Now, using Equation (20), we can say
[Equation]
Using the above two equations
[Equation]
we can say, for time t > t 1 ,
[Equation]
Unrolling the equation above
[Equation]
we get
[Equation]
Now, as γ u ( t ) → ∞ , therefore, using Equation (20), we can say
[Equation]
Now, using this identity, along with the assumption (A5), Equation (21) and Lemma 5, we can say
[Equation]
Using this along with Equation (22) and Lemma 4, we can say
[Equation]
However, this is not possible as the vectors on LHS have bounded norm.
This contradicts.
Thus ∆ = 0 .
Case 2:
[Equation]
We are going to show that its not possible to have (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) g u (cid:107) = 0 .
By Lemma 3, we know there exists atleast one node s satisfying (cid:107) (cid:101) w s (cid:107) > 0 and (cid:107) (cid:101) g s (cid:107) > 0 .
Now from Equation (19), we can say
[Equation] [Equation]
Now, γ s ( t ) either diverges to ∞ or −∞ .
In both the cases, it is a strictly monotonic sequence for large enough t .
Also lim t →∞
[Equation]
as (cid:107) (cid:101) g u (cid:107) = 0 , (cid:107) (cid:101) g s (cid:107) > 0 and from Case 1 , w s ( t ) and −∇ w s L ( w ( t )) eventually get aligned.
Thus, using Stolz-Cesaro theorem, we can say
[Equation]
However, this is not possible as (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) w s (cid:107) > 0 .
This contradicts.
Therefore, this is not possible.
D P ROOF OF T HEOREM
Theorem.
Under assumptions (A1)-(A4) for gradient ﬂow and (A1)-(A5) for gradient descent, the following hold
(i) for EWN, (cid:107) (cid:101) w u (cid:107) > 0 , (cid:107) (cid:101) w v (cid:107) > 0 =
⇒ lim
t →∞ (cid:107) w u ( t ) (cid:107)(cid:107)∇ w u L ( w ( t )) (cid:107) (cid:107) w v ( t ) (cid:107)(cid:107)∇ w v L ( w ( t )) (cid:107) =(ii) for SWN, (cid:107) (cid:101) w u (cid:107) > 0 ,
[Table]
Proof for different cases will be split into different subsections and the corresponding case will be restated there for ease of the reader.
D. E XPONENTIAL W EIGHT N ORMALIZATION D.1. G RADIENT F LOW
Theorem.
Under assumptions (A1)-(A4) for gradient ﬂow, the following holds for EWN:
[Equation] [Equation]
Proof.
Consider u and v such that (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) w v (cid:107) > 0 .
Using Proposition 2, we can say (cid:107) (cid:101) g u (cid:107) > 0 and (cid:107) (cid:101) g v (cid:107) > 0 .
Also, from Proposition 2, we can say, for both u and v , weights and gradients converge in opposite directions.
Hence there exists a time t 2 , such that for any t > t 2 ,
• −∇ w u L ( w ( t )) and w u ( t ) make an angle (cid:15) or lesser with each other • −∇ w v L ( w ( t )) and w v ( t ) make an angle (cid:15) or lesser with each other
Then, using Equation (7), we can say for any time t > t 2 , (cid:107) w u ( t ) (cid:107) ≥ (cid:107) w u ( t 2 ) (cid:107) + cos ( (cid:15) ) (cid:90)
t k = tη ( k ) (cid:107) w u ( k ) (cid:107)
(cid:107)∇ w u L ( w ( k ))
(cid:107) dk (cid:107) w u ( t ) (cid:107) ≤ (cid:107) w u ( t 2 ) (cid:107) + (cid:90) t
k
=
tη
( k ) (cid:107) w u ( k ) (cid:107)
(cid:107)∇ w u L ( w ( k ))
(cid:107) dk (cid:107) w v ( t ) (cid:107) ≥ (cid:107) w v ( t 2 ) (cid:107) + cos ( (cid:15) )
(cid:90) t k = tη ( k ) (cid:107) w v ( k ) (cid:107)
(cid:107)∇ w v L ( w ( k ))
(cid:107) dk (cid:107) w v ( t ) (cid:107) ≤ (cid:107) w v ( t 2 ) (cid:107) + (cid:90) t
k
=
tη
( k ) (cid:107) w v ( k ) (cid:107)
(cid:107)∇ w v L ( w ( k ))
(cid:107) dk
Using the above equations, we can say, for time t > t 2 ,
[Equation]
We know that both integrals diverge as (cid:107) w u ( t ) (cid:107) and (cid:107) w v ( t ) (cid:107) → ∞ , lim t →∞ (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) and lim t →∞
[Table]
exist.
Taking limit t → ∞ on both the equations and using the Integral form of Stolz-Cesaro theorem, we get lim t →∞
[Equation]
As we know these limits exist and this holds for any (cid:15) >
[Equation]
therefore
[Equation]
Simplifying it further
[Equation]
we get
[Equation]
D.1. G RADIENT D ESCENT
Theorem.
Under assumptions (A1)-(A5) for gradient descent, the following holds for EWN:
[Equation]
Proof.
Consider two nodes u and s such that (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) w s (cid:107) > 0 .
From Proposition2, we know (cid:107) (cid:101) g u (cid:107) > 0 , (cid:107) (cid:101) g s (cid:107) > 0 and for both u and s , weights and gradients eventually get aligned opposite to each other.
Using Equation (11), we can say
[Equation]
Thus, α u ( t ) − α s ( t ) =
( α u (0) − α s (0))+
[Equation]
Now, we know, α u ( t ) and α s ( t ) → ∞ .
Also, as (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) w s (cid:107) > 0 , therefore α u ( t ) − α s ( t ) converges.
Therefore the RHS of Equation (23) converges as well.
However, RHS is the difference of two diverging series.
Also, lim t →∞
[Equation]
exists.
Therefore, using Lemma 6, we can say
[Equation]
D. S TANDARD W EIGHT N ORMALIZATION D.2. G RADIENT F LOW
Theorem.
Under assumptions (A1)-(A4) for gradient ﬂow, the following holds for SWN:
[Equation]
Proof.
Consider u and v such that (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) w v (cid:107) > 0 .
Using Proposition 2, we can say (cid:107) (cid:101) g u (cid:107) > 0 and (cid:107) (cid:101) g v (cid:107) > 0 .
Also, from Proposition 2, we can say, for both u and v , weights and gradients converge in opposite directions.
Consider a time t 2 , such that for any t > t 2 ,
• −∇ w u L ( w ( t )) and w u ( t ) atmost make an angle (cid:15) with each other • −∇ w v L ( w ( t )) and w v ( t ) atmost make an angle (cid:15) with each other
Then, using Equation (16), we can say for any time t > t 2 ,
[Equation]
cos
[Equation]
) (cid:90)
[Equation]
k = tη
[Equation]
k
[Equation]
(cid:107)∇ w u L ( w ( k ))
(cid:107) dk (cid:107) w
[Equation]
t )
[Equation]
≤
[Equation]
Using the above equations
[Equation]
we can say
[Equation]
for time t > t 2 ,
[Equation]
L
[Equation]
w ( k ))
[Equation]
dk
We know that both integrals diverge as (cid:107) w u ( t ) (cid:107) and (cid:107) w v ( t ) (cid:107) → ∞ , lim t →∞ (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) and lim t →∞ (cid:107)∇ wu ( t ) L ( w ( t ))
(cid:107) (cid:107)∇ wv ( t ) L ( w ( t ))
(cid:107) exist.
Taking limit t → ∞ on both the equations and using the Integral form of Stolz-Cesaro theorem, we get
[Equation]
As we know these limits exist and this holds for any (cid:15) > 0 , therefore
[Equation]
D.2. G RADIENT D ESCENT
Theorem.
Under assumptions (A1)-(A5) for gradient descent, the following holds for SWN:
[Equation]
Proof.
Consider u and v such that (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) w s (cid:107) > 0 .
Using Proposition 2, we can say (cid:107) (cid:101) g u (cid:107) > 0 and (cid:107) (cid:101) g s (cid:107) > 0 .
Also, from Proposition 2, we can say, for both u and s , weights and gradients converge in opposite directions.
Now from Equation (19), we can say
[Equation]
Now, γ s ( t ) either diverges to ∞ or −∞ .
In both the cases, it is a strictly monotonic sequence for large enough t .
Also lim t →∞ γ u ( t +1) − γ u ( t ) γ s ( t +1)
− γ s ( t ) exists.
Therefore, using Stolz-Cesaro Theorem, we can say
[Equation]
E P ROOF OF P ROPOSITION
Proposition.
Let assumptions (A1)-(A5) be satisﬁed.
Consider two nodes u and v in the network such that (cid:107)
[Equation] g [Equation] ≥ (cid:107) (cid:101) g u (cid:107) > 0 and (cid:107) w u ( t ) (cid:107) , (cid:107) w v ( t ) (cid:107) → ∞ .
Let (cid:107) (cid:101)
g u (cid:107) (cid:107) (cid:101) g v (cid:107) be denoted by c .
Let (cid:15), δ be such that 0 < (cid:15) <
c and 0 < δ < 2 π .
Then, the following holds:
There exists a time t 1 , such that for all t > t 1 both SWN and EWN trajectories have the following properties: (a) (cid:107)∇ w u L ( w ( t ))
(cid:107) (cid:107)∇ w v L ( w ( t ))
(cid:107) ∈ [ c − (cid:15), c
+ (cid:15) ] (b) (cid:16) w u ( t ) (cid:107) w u ( t ) (cid:107) (cid:17) (cid:62) (cid:16) −∇ w u L ( w ( t ))
(cid:107)∇ w u L ( w ( t )) (cid:107) (cid:17) ≥ cos( δ ) (c) (cid:16) w v ( t ) (cid:107) w v ( t ) (cid:107) (cid:17) (cid:62) (cid:16) −∇ w v L ( w ( t ))
(cid:107)∇ w v L ( w ( t )) (cid:107) (cid:17) ≥ cos( δ ) .
for SWN, lim t →∞ (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) =
c 3.
for EWN, if at some time t 2 > t 1 , (a) (cid:107) w u ( t 2 ) (cid:107) (cid:107) w v ( t 2 ) (cid:107) > 1 ( c − (cid:15) )
cos( δ ) =
⇒ lim
t →∞ (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) =
∞ (b) (cid:107) w u ( t 2 ) (cid:107) (cid:107) w v ( t 2 ) (cid:107) < cos( δ )
[Equation]
(cid:15)
[Equation]
The proof of different cases will be split into multiple subsections and corresponding proposition will be stated there for ease of the reader.
E. S TANDARD W EIGHT N ORMALIZATION E.1. G RADIENT F LOW
Proposition.
Let assumptions (A1)-(A4) be satisﬁed.
Consider two nodes u and v in the network such that (cid:107) (cid:101) g v (cid:107) ≥ (cid:107) (cid:101) g u (cid:107) > 0 and (cid:107) w u ( t ) (cid:107) , (cid:107) w v ( t ) (cid:107) → ∞ .
Let (cid:107) (cid:101)
g u (cid:107) (cid:107) (cid:101) g v (cid:107) be denoted by c .
Let (cid:15), δ be such that 0 < (cid:15) <
c and 0 < δ < 2 π .
Then, the following holds:
There exists a time t 1 , such that for all t > t 1 , SWN trajectory has the following properties: (a) (cid:107)∇ w u L ( w ( t ))
(cid:107) (cid:107)∇ w v L ( w ( t ))
(cid:107) ∈ [ c − (cid:15), c
+ (cid:15) ] (b) (cid:16) w u ( t ) (cid:107) w u ( t ) (cid:107) (cid:17) (cid:62) (cid:16) −∇ w u L ( w ( t ))
(cid:107)∇ w u L ( w ( t )) (cid:107) (cid:17) ≥ cos( δ ) (c) (cid:16) w v ( t ) (cid:107) w v ( t ) (cid:107) (cid:17) (cid:62) (cid:16) −∇ w v L ( w ( t )) (cid:107)∇ w v L ( w ( t )) (cid:107) (cid:17) ≥ cos( δ ) . lim t →∞ (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) = [Equation]
Proof.
The proof of part 1a, i.e, (cid:107)∇ w [Equation] ( w ( t ))
[Equation]
v L ( w ( t ))
[Equation]
∈ [ c − (cid:15), c + (cid:15) ] follows from the deﬁnition of limit as (cid:107)∇
[Equation]
L
[Equation]
w
[Equation]
t )
[Equation]
t )
[Equation]
(cid:107) tends to c .
Now, we will move to the proof of part 1b, i.e., (cid:16) w u ( t ) [Equation] [Equation] w u L ( w ( t ))
(cid:107) (cid:17) ≥ cos( δ ) .
The assumptions in this Proposition differ slightly from Proposition 2 and thus the proof is slightly more involved as we also need to show that w u ( t ) converges in direction.
The proof will be given for γ u → ∞ .
The one for γ u → −∞ can be handled similarly.
As (cid:107) (cid:101) g u (cid:107) > 0 , therefore ∇ w u L ( w ( t )) converges in direction.
Therefore, for every τ satisfying 0 < τ < 2 π , there exists a time t 3 , such that for t > t 3 , (cid:16) −∇
[Equation]
L
[Equation]
w
[Equation]
t )) [Equation] [Equation] g u
[Equation]
≥ cos( τ ) . Now, Let’s assume that w u ( t ) does not converge in the direction of (cid:101) g u . Then, there must exist a τ satisfying 0 < τ < 2 π , such that for this τ , there exists a time t 4 > t 3 satisfying [Equation]
cos(
[Equation]
where ∆ > τ . Now, we are going to show that for any κ satisfying τ < κ < ∆ , there exists a time t 5 > t 4 such that v u ( t 5 ) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) > cos( κ ) . Let’s say for a given κ , no such t 5 exists. Then, taking dot product with (cid:101) g u (cid:107) (cid:101) g u (cid:107) on both sides of Equation (17), we can say (cid:18) (cid:101) g
[Equation]
) (cid:18) [Equation] ( t )) (cid:107)∇
[Equation]
L ( w ( t )) (cid:107) (cid:19) Now, as (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) (cid:62) (cid:16) −∇ w u L ( w ( t )) (cid:107)∇ w u L ( w ( t )) (cid:107) (cid:17) ≥ cos( τ ) and (cid:16) (cid:101) g u cid:107) (cid:101) g u (cid:107) (cid:17) (cid:62) v u ≤ cos( κ ) , we can say (cid:18) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:19) (cid:62) d
[Equation]
dt [Equation] Now, using the fact that γ u → ∞ and using Equation (16), we can say [Equation]
= ∞
Using this fact and integrating the Equation (24) on both the sides from t 4 to ∞ , we get a contradiction as vectors on LHS have a ﬁnite norm while RHS tends to ∞ .
Thus, for every κ between τ and ∆ , there must exist a t 5 , such that v u ( t 5 )
[Equation]
(cid:16)
[Equation]
> cos( κ ) .
Now, we are going to show for all t ≥ t 5 , v u ( t ) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) > cos( κ ) .
Now, consider any β such that τ < β < κ .
Using similar argument as in Equation (24), we can say, if for any t 6 > t 5 , v u ( t 6 ) (cid:62) (cid:16) [Equation] g [Equation]
< cos( β ) , then
[Equation]
This means that the dot product between (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) and v u ( t ) goes up, whenever (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) (cid:62) v u ( t ) < cos( τ ) .
Therefore, its not possible that v u ( t ) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) ≤ cos( κ ) arbitrarily chosen between τ and ∆ , and the argument holds for any (cid:15) > 0 , w u ( t ) converges in the direction of (cid:101) g u
The proof of part 1c, i.e, (cid:16) w v ( t ) (cid:107) w v ( t ) (cid:107) (cid:17) (cid:62) (cid:16) −∇ w v L ( w ( t )) (cid:107)∇ w v L ( w ( t )) (cid:107) (cid:17) ≥ cos( δ ) can be shown in the same way as 1b. The proof of part 2, i.e, lim t →∞ (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) = c can be shown in the same way as Theorem 2 for SWN gradient ﬂow from Appendix D.2.1.
E.1. G RADIENT D ESCENT
Proposition.
Let assumptions (A1)-(A5) be satisﬁed.
Consider two nodes u and v in the network such that (cid:107) (cid:101) g v (cid:107) ≥ (cid:107) (cid:101) g u (cid:107) > 0 and (cid:107) w u ( t ) (cid:107) , (cid:107) w v ( t ) (cid:107) → ∞ .
Let (cid:107) (cid:101)
g u (cid:107) (cid:107) (cid:101) g v (cid:107) be denoted by c .
Let (cid:15), δ be such that 0 < (cid:15) <
c and 0 < δ < 2 π .
Then, the following holds:
There exists a time t 1 , such that for all t > t 1 , SWN trajectory has the following properties: (a) (cid:107)∇ w u L ( w ( t ))
(cid:107) (cid:107)∇ w v L ( w ( t ))
(cid:107) ∈ [ c − (cid:15), c
+ (cid:15)
[Table]
Proof.
The proof of part 1a, i.e, (cid:107)∇ w u L ( w ( t ))
(cid:107) (cid:107)∇ w v L ( w ( t )) (cid:107) ∈ [ c − (cid:15), c + (cid:15) ] follows from the deﬁnition of limit as (cid:107)∇ w u L ( w ( t )) (cid:107) (cid:107)∇ w v L ( w ( t ))
(cid:107) tends to c . Now, we will move to the proof of part 1b, i.e, (cid:16) w u ( t ) (cid:107) w u ( t ) (cid:107) (cid:17) [Table] (cid:16) −∇ w u L
[Table]
w
[Table] [Table]
w u L ( w ( t ))
(cid:107) (cid:17) ≥ cos( δ ) .
The assumptions in this Proposition differ slightly from Proposition 2 and thus the proof is slightly more involved as we also need to show that w u ( t ) converges in direction.
The proof will be given for γ u → ∞ .
The one for γ u → −∞ can be handled similarly.
As (cid:107) (cid:101) g u (cid:107) > 0 , therefore ∇ w u L ( w ( t )) converges in direction.
Therefore, for every τ satisfying 0 < τ < 2 π , there exists a time t 3 , such that for t > t 3 , (cid:16)
[Equation]
≥ cos( τ ) .
Now, Let’s assume that w u ( t ) does not converge in the direction of (cid:101) g u .
Then, there must exist a τ satisfying 0 < τ < 2 π , such that for this τ , there exists a time t 4 > t 3 satisfying
[Equation]
, where ∆ > τ .
Now, we are going to show that for any κ satisfying τ < κ < ∆ , there exists a time t 5 > t 4 such that (cid:16) v u ( t 5 ) (cid:107) v u ( t 5 ) (cid:107) (cid:17) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) > cos( κ ) .
Let’s say for a given κ , no such t 5 exists.
Then, taking dot product with (cid:101) g u (cid:107) (cid:101) g u (cid:107) on both sides of Equation (20), we can say v u ( t + 1) (cid:62) (cid:101) g u (cid:107) (cid:101) g u (cid:107) =
[Equation]
(cid:107)
[Equation]
+ η ( t ) γ u ( t ) (cid:107) v u ( t ) (cid:107)(cid:107)∇ w u L ( w ( t ))
(cid:107) (cid:18) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:19) (cid:62) ( I − v u ( t ) v u ( t ) (cid:62) (cid:107) v u ( t ) (cid:107) 2 ) (cid:18) −∇ w u L ( w ( t ))
(cid:107)∇ w u L ( w ( t )) (cid:107) (cid:19)
Now, as (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) (cid:62) (cid:16) −∇ w u L ( w ( t )) (cid:107)∇ w u L ( w ( t ))
(cid:107) (cid:17) ≥ cos( τ ) and (cid:16) (cid:101) g u
cid:107) (cid:101) g u (cid:107) (cid:17) (cid:62) (cid:16) v u ( t ) (cid:107) v u ( t ) (cid:107) (cid:17) ≤ cos( κ ) , we can say v u ( t + 1) (cid:62) (cid:101) g u (cid:107) (cid:101) g u (cid:107) ≥ v u ( t ) (cid:62) (cid:101) g u (cid:107) (cid:101) g u (cid:107) +
(cos( τ ) − cos( κ ))( η ( t ) γ u ( t ) (cid:107) v u ( t ) (cid:107) (cid:107)∇ w u L ( w ( t )) (cid:107) ) (26)
Now, using arguments similar to the proof of Proposition 2 for SWN gradient descent in Appendix
C.2.2, we can show that the above statement leads to a contradiction and thus there must exist a tsuch that (cid:16)
[Equation]
( t
[Equation]
(cid:17)
[Equation]
(cid:17) > cos( κ ) .
Now, we are going to show that there exists a t 6 > t 5 , such that for all t > t 6 , (cid:16)
[Table]
> cos( κ ) .
Consider a β such that τ < β < κ .
Now, if at any time t, (cid:16)
[Table]
[Equation]
< cos( β ) , then, similar to Equation (26), we can say
[Equation]
Using the upper bound on (cid:107) v u ( t + 1) (cid:107) from Equation (21), we can say
[Equation]
[Equation]
Let η ( t ) γ u ( t ) (cid:107) v u ( t ) (cid:107) (cid:107)∇ w u L ( w ( t ))
(cid:107) be denoted by χ ( t ) .
Then, the above equation can be rewritten as
[Equation]
Now
[Equation]
we are going to show that for a small enough χ ( t ) , RHS is greater than v
[Table]
v
[Table]
{+( t ) (cid:62) (cid:101) g u (cid:107) v u ( t ) (cid:107)(cid:107) (cid:101) g u (cid:107) (cid:107) v u ( t ) (cid:107) (cid:112) (cid:107) v u ( t ) (cid:107) 
+}
+ χ ( t ) 2 + (cos( τ ) − cos( β )) χ
( t ) (cid:112) (cid:107) v u ( t ) (cid:107) 2 + χ ( t ) 2 > v u ( t ) (cid:62) (cid:101) g u (cid:107) v u ( t ) (cid:107)(cid:107) (cid:101) g u (cid:107) =
⇒ (cos( τ ) − cos( β )) χ
{+( t ) (cid:112) (cid:107) v u ( t ) (cid:107) 2 + χ ( t ) 2 > v u ( t ) (cid:62) (cid:101) g u (cid:107) v u ( t ) (cid:107)(cid:107) (cid:101) g u (cid:107) (1 − (cid:107) v u ( t ) (cid:107) (cid:112) (cid:107) v u ( t ) (cid:107) 
+}
+ χ ( t ) 2 ) =
{+⇒ (cos( τ ) − cos( β )) > v u ( t ) (cid:62) (cid:101) g u (cid:107) v u ( t ) (cid:107)(cid:107) (cid:101) g u (cid:107) ( (cid:112) (cid:107) v u ( t ) (cid:107) 
+}
+ χ ( t ) 2 − (cid:107) v u ( t ) (cid:107) χ ( t ) )
Clearly as χ ( t ) → 0 , the RHS tends to 0, therefore the equation is satisﬁed. Thus for a small enough χ ( t ) , RHS of Equation (27) is greater than v u ( t ) (cid:62) (cid:101) g u (cid:107) v u ( t ) (cid:107)(cid:107) (cid:101) g u (cid:107) . As (cid:107) v u ( t ) (cid:107) keeps on increasing and by Assumption (A5), lim t →∞ η ( t ) γ u ( t ) (cid:107)∇ w u L ( w ( t ))
(cid:107) =
, we can say there exists a time t 7 , such that for any t > t 7 , v u ( t ) (cid:62) (cid:101) g u (cid:107) v u ( t ) (cid:107)(cid:107) (cid:101) g u (cid:107) goes up whenever (cid:16) v u ( t ) (cid:107) v u ( t ) (cid:107) (cid:17) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) < cos( β ) . Also, by using Equation (20) and Assumption (A5), we can say, that there exists a time t 8 , such that for t > t 8 , (cid:16) v [Equation] t
[Equation]
t [Equation]
(cid:101) [Equation] (cid:17) > cos( β ) = ⇒ (cid:16)
[Equation]
> cos( κ ) , as the RHS of Equation (20) goes to 0 norm in limit. Now, deﬁne t 6 > max ( t 7 , t 8 ) such that [Equation] > cos( κ ) (must exist from previous arguments).
Then, as the dot product always goes up when between cos( β ) and cos( κ ) , and can’t go in a single step from being greater than cos( β ) to less than cos( κ ) , therefore, for every t > t 6 , (cid:16)
[Equation]
> cos( κ ) .
Now as the above argument holds for any κ between τ and ∆ , and for any τ > 0 , we can say that w u ( t ) converges in direction of (cid:101) g u . The proof of part 1c, i.e, (cid:16) w v ( t ) (cid:107) w v ( t ) (cid:107) (cid:17) (cid:62) (cid:16) [Equation]
t
[Equation] t ) [Equation] (cid:17) ≥ cos( δ ) , follows exactly the same steps as part 1b.
The proof of part 2, i.e, lim t →∞ (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) =
c can be shown in the same way as the proof of Theorem 2 for SWN gradient descent from Appendix D.2.2.
E. E XPONENTIAL W EIGHT N ORMALIZATION E.2. G RADIENT F LOW
Proposition.
Let assumptions (A1)-(A4) be satisﬁed.
Consider two nodes u and v in the network such that (cid:107) (cid:101) g v (cid:107) ≥ (cid:107) (cid:101) g u (cid:107) > 0 and (cid:107) w u ( t ) (cid:107) , (cid:107) w v ( t ) (cid:107) → ∞ .
Let (cid:107) (cid:101)
g u (cid:107) (cid:107) (cid:101) g v (cid:107) be denoted by c .
Let (cid:15), δ be such that 0 < (cid:15) <
c and 0 < δ < 2 π .
Then, the following holds:
There exists a time t 1 , such that for all t > t 1 , EWN trajectory has the following properties: (a) (cid:107)∇ w u L ( w ( t ))
(cid:107) (cid:107)∇ w v L ( w ( t ))
(cid:107) ∈ [ c − (cid:15), c
+ (cid:15) ] (b) (cid:16) w u ( t ) (cid:107) w u ( t ) (cid:107) (cid:17) (cid:62) (cid:16) −∇ w u L ( w ( t ))
(cid:107)∇ w u L ( w ( t )) (cid:107) (cid:17) ≥ cos( δ ) (c) (cid:16) w v ( t ) (cid:107) w v ( t ) (cid:107) (cid:17) (cid:62) (cid:16) −∇ w v L ( w ( t ))
(cid:107)∇ w v L ( w ( t )) (cid:107) (cid:17) ≥ cos( δ ) .
If at some time t 2 > t 1 , (a) (cid:107) w u ( t 2 ) (cid:107)
[Equation]
Proof.
The proof of part 1a, i.e, [Equation] w [Equation]
∈ [ c −
[Equation]
, c +
[Equation]
] follows from the deﬁnition of limit as [Equation] tends to c . Now, we will move to the proof of part 1b, i.e.,
[Equation]
≥ cos( δ ) . The assumptions in this Proposition differ slightly from Proposition 2 and thus the proof is slightly more involved as we also need to show that w u ( t ) converges in direction. As (cid:107) (cid:101) g u (cid:107) > 0 , therefore ∇ w u L ( w ( t )) converges in direction. Therefore, for every τ satisfying 0 < τ < 2 π , there exists a time t 3 , such that for t > t 3 , (cid:16)
[Equation] w ( t )
[Equation] w u L ( w ( t )) (cid:107) (cid:17) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) ≥ cos( τ ) .
Now, Let’s assume that w u ( t ) does not converge in the direction of (cid:101) g u . Then, there must exist a τ satisfying 0 < τ < 2 π , such that for this τ , there exists a time t 4 > t 3 satisfying v u ( t 4 ) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) = cos(∆) , where ∆ > τ . Now, we are going to show that for any κ satisfying τ < κ < ∆ , there exists a time t 5 > t 4 such that v u ( t 5 ) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) > cos( κ ) . Let’s say for a given κ , no such t 5 exists.
Then, taking dot product with (cid:101) g u (cid:107) (cid:101) g u (cid:107) on both sides of Equation (8), we can say [Equation] (cid:19) Now, as (cid:16)
[Equation]
g [Equation] (cid:16) −∇
[Equation] L [Equation]
t )) (cid:107)∇ w u L ( w ( t )) (cid:107) (cid:17) ≥ cos( τ ) and (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) (cid:62) v u ≤ cos( κ ) , we can say [Equation] g
[Equation]
Now, using the fact that α u → ∞ and using Equation (7), we can say [Equation] Using this fact and integrating the Equation (28) on both the sides from t 4 to ∞ , we get a contradiction as vectors on LHS have a ﬁnite norm while RHS tends to ∞ . Thus, for every κ between τ and ∆ , there must exist a t 5 , such that v u ( t 5 ) (cid:62) (cid:16) (cid:101) g u
[Equation]
(cid:101) g u (cid:107) (cid:17) > cos( κ ) . Now, we are going to show for all t ≥ t 5 , v u ( t ) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) > cos( κ ) . Now, consider any β such that τ < β < κ . Using similar argument as in Equation (28), we can say, if for any t 6 > t 5 , v u ( t 6 ) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) < cos( β ) , then
[Equation]
This means that the dot product between (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) and v u ( t ) goes up, whenever (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) (cid:62) v u ( t ) < cos( τ ) .
Therefore, its not possible that v u ( t ) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) ≤ cos( κ ) for any t > t 5 .
As κ can be arbitrarily chosen between τ and ∆ , and the argument holds for any (cid:15) > 0 , w u ( t ) converges in the direction of (cid:101) g u The proof of part 1c, i.e, (cid:16) w v ( t ) (cid:107) w v ( t ) (cid:107) (cid:17) (cid:62) (cid:16) −∇ w v L ( w ( t )) (cid:107)∇
[Equation]
( w ( t )) (cid:107) (cid:17) ≥ cos( δ ) follows exactly the same steps as part 1b. Now, we will move to the proof of part 2a, i.e, (cid:107) w u ( t 2 ) (cid:107) (cid:107) w v ( t 2 ) (cid:107) >
[Equation]
Using Equation (7), [Equation] Using the equation above and part 1 of the Proposition, we can say for t > t 1 ,
[Equation] In this case, using Equation (31), we can see d (cid:107) w u [Equation] t
[Equation]
w v ( t ) (cid:107) dt > 0 at t 2 . Thus, (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) always remains greater than 1 ( c − (cid:15) )cos( δ ) and keeps on increasing.
Let’s denote (cid:107) w u ( t 2 ) (cid:107) (cid:107) w v ( t 2 ) (cid:107) by ∆ .
Then we can say
[Equation]
As α u → ∞ , therefore using Equation (7), we can say (cid:82) ∞ t 2 η ( t ) (cid:107) w u ( t ) (cid:107)(cid:107)∇ w u L ( w ( t )) (cid:107) dt → ∞ .
Thus, integrating both the sides of the equation above from t 2 to ∞ , we get (cid:90) [Equation] dt
[Equation]
∞ Thus lim t →∞
[Equation]
w u [Equation] t )
[Equation]
(cid:107) w v ( t ) (cid:107) = ∞ . Now, we will move to the proof of part 2b, i.e, (cid:107) w u ( t 2 ) (cid:107) (cid:107) w v ( t 2 ) (cid:107) < cos( δ ) c + (cid:15) = ⇒ lim
t →∞
[Equation]
Using Equation (30) and part 1 of the Proposition, we can say for t > t 1 ,
[Equation]
In this case, using Equation (32), we can see d (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) dt < 0 at t 2 .
Thus, (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) always remains smaller than cos( δ ) c
+ (cid:15) and keeps on decreasing.
Now, lets say lim t →∞ (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) > 0 . This means that
[Equation]
w u
[Equation]
t )
[Equation]
w v ( t ) (cid:107) > ∆ , for some ∆ > 0 .
Also, let’s denote (cid:107) w u ( t 2 ) (cid:107) (cid:107) w v ( t 2 ) (cid:107) by β . Then we can say
[Equation]
As α v → ∞ , therefore using Equation (7), we can say (cid:82) ∞ t 2 η (cid:107)
[Equation]
t ) (cid:107)(cid:107)∇ w v L ( w ( t )) (cid:107) dt → ∞ .
Thus, integrating both the sides of the equation above from t 2 to ∞ , we get
[Equation]
This is not possible as (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) is lower bounded by 0. Thus lim t →∞ (cid:107) w u
[Equation]
w v [Equation] (cid:107) =
[Equation]
E.2. G RADIENT D ESCENT
Proposition.
Let assumptions (A1)-(A5) be satisﬁed.
Consider two nodes u and v in the network such that (cid:107) (cid:101) g v (cid:107) ≥ (cid:107) (cid:101) g u (cid:107) > 0 and (cid:107) w u ( t ) (cid:107) , (cid:107) w v ( t ) (cid:107) → ∞ .
Let (cid:107) (cid:101)
g u (cid:107) (cid:107) (cid:101) g v (cid:107) be denoted by c .
Let (cid:15), δ be such that 0 < (cid:15) <
c and 0 < δ < 2 π .
Then, the following holds:
There exists a time t 1 , such that for all t > t 1 , EWN trajectory has the following properties : (a) (cid:107)∇ w u L ( w ( t ))
(cid:107) (cid:107)∇ w v L ( w ( t ))
(cid:107) ∈ [ c − (cid:15), c
+ (cid:15) ] (b) (cid:16) w u ( t ) (cid:107) w u ( t ) (cid:107) (cid:17) (cid:62) (cid:16) −∇ w [Figure] ( w ( t ))
[Figure]
w ( t )) (cid:107) (cid:17) ≥ cos( δ ) (c) (cid:16) w v ( t ) (cid:107) w v ( t ) (cid:107) (cid:17) (cid:62) (cid:16) −∇ w v L ( w ( t )) (cid:107)∇ w v L ( w ( t )) (cid:107) (cid:17) ≥ cos( δ ) . If at some time t 2 > t 1 , (a) (cid:107) w u ( t 2 ) (cid:107) (cid:107) w v ( t 2 ) (cid:107) >
[Equation] [Equation]
(b) (cid:107) w u ( t 2 ) (cid:107) (cid:107) w v ( t 2 ) (cid:107) < cos( δ ) c + (cid:15) [Equation] u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107)
[Equation]
Proof.
The proof of part 1a, i.e, (cid:107)∇ w u L ( w ( t )) (cid:107) (cid:107)∇ w v L ( w ( t )) (cid:107) ∈ [ c − (cid:15), c
+ (cid:15) ] follows from the deﬁnition of limit as (cid:107)∇ w u L ( w ( t )) (cid:107) (cid:107)∇ w v L ( w ( t )) (cid:107) tends to c . Now, we will move to the proof of part 1b, i.e, (cid:16) w u ( t ) (cid:107) w u ( t ) (cid:107) (cid:17) (cid:62) (cid:16) −∇ w u L ( w ( t ))
(cid:107)∇ w u L ( w ( t )) (cid:107) (cid:17) ≥ cos( δ ) .
The assumptions in this Proposition differ slightly from Proposition 2 and thus the proof is slightly more involved as we also need to show that w u ( t ) converges in direction. As (cid:107) (cid:101) g u (cid:107) > 0 , therefore ∇ w u L ( w ( t )) converges in direction. Therefore, for every τ satisfying 0 < τ < 2 π , there exists a time t 3 , such that for t > t 3 , (cid:16) −∇ w [Equation]
L ( w ( t )) (cid:107)∇ w
[Equation]
L ( w ( t )) (cid:107) (cid:17) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) ≥ cos( τ ) .
Now, Let’s assume that w u ( t ) does not converge in the direction of (cid:101) g u .
Then, there must exist a τ satisfying 0 < τ < 2 π , such that for this τ , there exists a time t 4 > t 3 satisfying v u ( t 4 ) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) = cos(∆) , where ∆ > τ .
Now, we are going to show that for any κ satisfying τ < κ < ∆ , there exists a time t 5 > t 4 such that (cid:16) v u ( t 5 ) (cid:107) v u ( t 5 ) (cid:107) (cid:17) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) > cos( κ ) .
Let’s say for a given κ , no such t 5 exists.
Then, taking dot product with (cid:101) g u (cid:107) (cid:101) g u (cid:107) on both sides of Equation (12), we can say
[Equation]
Now, as (cid:16) (cid:101) g
[Equation]
)) (cid:107) (cid:17) ≥ cos( τ ) and (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) (cid:62) (cid:16) v u ( t ) (cid:107) v u ( t ) (cid:107) (cid:17) ≤ cos( κ ) , we can say
[Equation]
Now, using arguments similar to the proof of Proposition 2 for EWN gradient descent in Appendix
C.1.2, we can show that the above statement leads to a contradiction and thus there must exist a tsuch that (cid:16) v u ( t 5 ) (cid:107) v u ( t 5 ) (cid:107) (cid:17) (cid:62) (cid:16) (cid:101) g u (cid:107) (cid:101) g u (cid:107) (cid:17) > cos( κ ) .
Now, we are going to show that there exists a t 6 > t 5 , such that for all t > t 6 , (cid:16)
[Equation]
(cid:16)
[Equation] (cid:101) g u (cid:107) [Equation] > cos( κ ) .
Consider a β such that τ < β < κ . Now, if at any time t, (cid:16) v u [Equation] t
[Equation]
v u
[Equation]
(cid:17) < cos( β ) , then, similar to Equation (33), we can say
[Equation]
Using the upper bound on (cid:107) v u ( t + 1) (cid:107) from Equation (13), we can say
[Equation]
[Equation]
Let η ( t ) e αu ( t ) (cid:107) v u ( t ) (cid:107) (cid:107)∇ w u L ( w ( t )) (cid:107) be denoted by χ ( t ) . Then, the above equation can be rewritten as [Equation]
Now
[Equation]
we are going to show that for a small enough χ ( t ) , RHS is greater than v u ( t ) [Equation] = ⇒ [Equation]
cos
[Equation]
[Equation]
= ⇒
[Equation]
cos
[Equation]
−
[Equation]
>
[Equation]
Clearly as χ ( t ) → 0 , the RHS tends to 0, therefore the equation is satisﬁed.
Thus for a small enough χ ( t ) , RHS of Equation (34) is greater than v u ( t ) (cid:62) (cid:101) g u (cid:107) v u ( t ) (cid:107)(cid:107) (cid:101) g u (cid:107) .
As (cid:107) v u ( t ) (cid:107) keeps on increasing and by
Assumption (A5), lim t →∞ η ( t ) γ u ( t ) (cid:107)∇ w u L ( w ( t ))
(cid:107) =
, we can say there exists a time t 7 , such that for any t > t 7 , v u
[Table]
goes up whenever (cid:16)
[Table] < cos( β ) . Also, by using Equation (12) and Assumption (A5), we can say, that there exists a time t 8 , such that for t > t 8 , (cid:16) v u ( t ) (cid:107) v u ( t ) (cid:107) (cid:17) [Equation]
g
[Equation]
g u
[Equation]
> cos( β )
[Equation]
> cos( κ ) , as the RHS of Equation (12) goes to 0 norm in limit.
Now, deﬁne t 6 > max ( t 7 , t 8 ) such that (cid:16)
[Equation]
g u
[Equation]
(cid:17) > cos( κ ) (must exist from previous arguments).
Then, as the dot product always goes up when between cos( β ) and cos( κ ) , and can’t go in a single step from being greater than cos( β ) to less than cos( κ ) , therefore, for every t > t 6 , (cid:16) v
[Table]
Now as the above argument holds for any κ between τ and ∆ , and for any τ > 0 , we can say that w u ( t ) converges in direction of (cid:101) g u .
The proof of part 1c, i.e, (cid:16) w v ( t ) (cid:107) w
[Equation] [Equation] ) (cid:107) (cid:17) (cid:62) (cid:16) −∇ w v L ( w ( t )) ( cid:107)∇ w v L ( w ( t ))
(cid:107) (cid:17) ≥ cos( δ ) follows exactly the same steps as part 1b. Now, we will move to the proof of part 2a, i.e, (cid:107) w u ( t 2 ) (cid:107) (cid:107) w v ( t 2 ) (cid:107) > [Equation] Using Equation (11) and part 1 of the Proposition, we can say
[Equation]
Thus, (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) keeps on increasing for t > t 2 .
It can either diverge to inﬁnity or converge to a ﬁnite value.
If it converges to a ﬁnite value, then by Stolz Cesaro theorem,
[Equation]
However, this is not possible as (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) > 1 c for every t > t 2 .
Thus, (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) diverges to inﬁnity.
Now, we will move to the proof of part 2b, i.e,
[Equation]
Using Equation (11) and part 1 of the Proposition, we can say
[Equation]
w v ( t
[Equation]
[Equation]
Thus, (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) keeps on decreasing for t > t 2 .
As it is always greater than zero, it must converge.
Therefore, by Stolz Cesaro Theorem, lim
[Equation]
For
[Equation] w [Equation] < 1 c , this can only be satisﬁed when lim t →∞ (cid:107) w u ( t ) (cid:107) (cid:107) w v ( t ) (cid:107) = 0 .
F P ROOF OF C OROLLARY
Corollary.
Consider a weight normalized(SWN or EWN) multilayer linear net, represented by y = W n W n − 1 ...
W 1 x .
Under assumptions (A1)-(A4) for gradient ﬂow and (A1)-(A5) for gradient descent, if the dataset is sampled from a continuous distribution w.r.t R d , then, with probability 1, θ = W (cid:62) 1 W (cid:62) 2 ....W (cid:62) n converges in direction to the maximum margin separator for all linearly separable datasets.
Proof.
Consider a linear net given by f = W n W n − 1 ...
W 1 x , where f is a scalar as we are considering a binary classiﬁcation problem with exponential loss.
Then
[Equation]
Now, atleast one of the neurons in every layer would have a non-zero component on (cid:101) w , otherwise Φ( (cid:101) w , x i )
=
for all i , that implies ρ = 0 .
Let u be one of the nodes of W 1 that has a non-zero component in (cid:101) w u and let w u be the k th row of the matrix W 1 .
Now, from Proposition 2, we know (cid:101) w u = λ (cid:101) g u .
Denoting the component of (cid:101) w along matrix W k by (cid:102) W k and using Equation (35), we get, for some λ k > 0 ,
[Equation]
where (( (cid:102) W n (cid:102) W n − 1 ..... (cid:102) W 2 ) (cid:62) )
[ k ] represents the k th component of the product column vector.
Let S represent the set of rows in W 1 that have a non-zero component in (cid:101) w .
Also, let (cid:101) θ denote the ﬁnal convergent direction of θ .
Then, for some µ > 0 , we can say
[Equation]
Thus, θ satisﬁes the KKT conditions of the maximum margin problem and if data is sampled from a continuous distribution w.r.t R d , then with probability 1(Soudry et al., 2018), would converge to the maximum margin separator.
G P ROOF OF T HEOREM
Theorem.
For EWN, under Assumptions (A1)-(A5) and lim t →∞ (cid:107) r ( t +1) − r ( t ) (cid:107) g
( t +1) − g ( t ) = 0 , the following hold
(cid:107) w ( t ) (cid:107) asymptotically grows at Θ (cid:16) (log( d ( t ))
1 L (cid:17) 2.
L ( w ( t )) asymptotically goes down at the rate of Θ (cid:16) 1 d ( t )(log d ( t ))(cid:17) .
First, we will establish rates for gradient ﬂow and then go to the case of gradient descent.
G. G RADIENT F LOW
Although the asymptotic convergence rates for smooth homogeneous neural nets have been established in Lyu & Li (2020), the proof technique becomes easier to understand for smooth homogeneous nets, without weight normalization.
G.1. U NNORMALIZED N ETWORK
Theorem.
For Unnorm, under Assumptions (A1)-(A4) for gradient ﬂow and lim t →∞ (cid:107) d r ( t ) dt (cid:107) g (cid:48) ( t ) = 0 , the following hold
(cid:107) w ( t ) (cid:107) asymptotically grows at Θ (cid:16) (log( t ) 1 L (cid:17) 2.
L ( w ( t )) asymptotically goes down at the rate of Θ (cid:18)
[Equation]
t
[Equation]
(cid:19) .
Proof.
Consider w = g ( t ) (cid:101) w + r ( t ) , where lim t →∞ (cid:107) r ( t ) (cid:107) g ( t ) = 0 and r ( t ) (cid:62) (cid:101) w = 0 .
Now, we make an additional assumption that lim t →∞ (cid:107) d r ( t ) d ( t ) (cid:107) g (cid:48) ( t )
= 0 .
This basically avoids any oscillations in r ( t ) for large t , where it can have a higher derivative, but the value may be bounded.
Now, we know
[Equation]
Now, we know (cid:107) d w ( t ) dt (cid:107) (cid:54) = 0 for any ﬁnite t , otherwise w won’t change and L can’t converge to 0.
Thus, for all t , we can say
[Equation]
Taking limit t → ∞ on both the sides, we get
[Equation]
Now, we know
[Equation]
Let S
= { i : Φ( (cid:101) w , x i ) =
min j Φ( (cid:101) w , x j ) } .
Then as ρ > 0 , we can say
[Equation]
where k is some constant.
Also, by the assumption
[Equation]
Substituting the above two equations in Equation (36), we get
[Equation]
k
Now, as loss goes down at the rate of e − ρg ( t ) L , multiplying the numerator and denominator by ρLg ( t ) L − 1 and denoting h ( t )
= ρg ( t ) L , we get
[Equation]
Thus, asymptotically, h ( t ) grows at Θ(log( t ) + (2 − 2 L ) log log t ) and thus loss goes down at Θ( t (log t )
2 − 2 L ) .
G.1. E XPONENTIAL W EIGHT N ORMALIZATION
Theorem.
For EWN, under Assumptions (A1)-(A4) for gradient ﬂow and lim t →∞ (cid:107) d r [Equation] t ) dt
[Equation]
g [Equation] t ) = 0 , the following hold (cid:107) w ( t ) (cid:107) asymptotically grows at Θ (cid:16) (log( t ) 1 L (cid:17) 2.
L ( w ( t )) asymptotically goes down at the rate of Θ (cid:16) 1 t (log t )(cid:17) .
Proof.
Consider w = g ( t ) (cid:101) w + r ( t ) , where lim t →∞ (cid:107) r ( t ) (cid:107) g ( t ) = 0 and r ( t ) (cid:62) (cid:101) w = 0 .
Now, we make an additional assumption that lim t →∞
[Equation]
d r [Equation] t [Equation]
d [Equation] t )
[Equation]
( t [Equation]
= 0 . In this case, [Equation]
However, in this case, for a node u ,
[Equation]
Consider a vector a ( t ) of equal dimension as w , and its components corresponding to a node u is given by a u ( t )
= −(cid:107) (cid:101) w u (cid:107) 2 d L ( w ( t ))
d w u .
Now as we know w converges in direction to (cid:101) w , therefore, using the update equation above, we can say
[Equation]
Using the update equation for − d L ( w ( t ))
d w , we can say
[Equation]
where k 1 is some constant.
Now, using the expression for a ( t ) , we can say
[Equation]
where k is some constant.
Using the equations above, we can say
[Equation]
Now, as loss goes down at the rate of e − ρg ( t ) L , multiplying the numerator and denominator by ρLg ( t ) L − 1 and denoting h ( t )
= ρg ( t ) L , we get
[Equation]
Thus, asymptotically, h ( t ) grows at Θ(log( t ) + 2 log log t ) and thus loss goes down at the rate of Θ( 1 t (log t ) 2 ) .
G. G RADIENT D ESCENT
Theorem.
For Exponential Weight Normalization, under Assumptions (A1)-(A5) and lim t →∞ (cid:107) r ( t +1) − r ( t ) (cid:107) g
( t +1) − g ( t ) = 0 , the following hold
(cid:107) w ( t ) (cid:107) asymptotically grows at Θ (cid:16) (log( d ( t ))
1 L (cid:17) 2.
L ( w ( t )) asymptotically goes down at the rate of Θ (cid:16) 1 d ( t )(log d ( t ))(cid:17) .
Proof.
Consider w = g ( t ) (cid:101) w + r ( t ) , where lim t →∞ (cid:107) r ( t ) (cid:107) g ( t ) = 0 and r ( t ) (cid:62) (cid:101) w = 0 .
Now, we make additional assumptions that lim t →∞ (cid:107) r ( t +1) − r [Equation] t )
[Equation]
g ( t +1) − g ( t ) = 0 . Consider a node u in the network that has (cid:107) (cid:101) w u (cid:107) > 0 .
The update equations for v u ( t ) and α u ( t ) are given by
[Equation]
Now, we will ﬁrst estimate (cid:107) e α u ( t +1) v u ( t +1) (cid:107) v u ( t +1)
(cid:107) −
e α u ( t ) v u ( t ) (cid:107) v u ( t ) (cid:107) (cid:107) . Let δ u ( t ) denote η ( t ) e α u ( t ) (cid:107)∇ w u L ( w ( t )) (cid:107) and (cid:15) u ( t ) denote the angle between v u ( t ) and −∇ w u L ( w ( t )) .
We know lim t [Equation] δ u [-( t ) = 
-][Equation]
{+t ) = 
+} and lim t →∞ (cid:15) u ( t )
= 0 .
Now, rewriting update equations in terms of these symbols, we get
[Equation]
where −∇ w u L ( w ( t )) v u ( t ) ⊥ denotes the component of −∇ w u L ( w ( t )) perpendicular to v u ( t ) .
Now, using these equations we can say
[Equation]
Now as lim
[Equation]
v u
[Equation]
t +1)
[Equation]
= 1 , therefore we can say
[Equation]
Now, as (cid:107) v u ( t ) (cid:107) keeps on increasing during the gradient descent trajectory, therefore we can say
[Equation]
v u ( t ) (cid:107)(cid:107) v u ( t +1) (cid:107) ≤ k , where k > 0 is some constant.
Now dividing both sides of Equation (38) by e α u ( t ) δ u ( t ) cos( (cid:15) u ( t )) and analyzing the coefﬁcient of the second term on RHS, we get
[Equation]
Taking norm on both sides of Equation (38), using Pythagoras theorem and the limits established above, we can say
[Equation]
Now, we also know
[Equation]
Now, using equations above and Equation (37), we can say
[Equation]
where c is some constant.
This determines the asymptotic rate of g ( t ) .
To get a better closed form, deﬁne deﬁne a map d : N → R , given by d ( t )
= (cid:80) tτ =
0 η ( τ ) and a real analytic function f ( t ) satisfying f ( d ( t ))
=
g ( t ) for all t ∈ N and lim t →∞ η ( t ) f
[Equation]
f [Equation]
= 0 .
Substituting this f in the equation above, we can say
[Equation]
Thus f ( d ( t )) grows at Θ( log ( d ( t )) 1 L ) .
Now, to get convergence rate for loss, multiply and divide the equation by ρf ( d ( t ))
L − 1 and denoting h ( d ( t ))
=
ρf ( d ( t ))
L , we get
[Equation]
Thus, h ( d ( t )) grows at the Θ(log( d ( t ))
+ 2 log log d ( t )) .
Now as g ( t ) =
f ( d ( t )) and ρg ( t ) L = h ( d ( t )) , therefore g ( t ) asymptotically grows at Θ(log d ( t ) 1 L ) and loss goes down asymptotically at Θ( 1 d ( t )log d ( t ) 2 ) .
H C ROSS -E NTROPY L OSS
In this section, we will provide the corresponding assumptions and theorems, along with their proofs, for cross-entropy loss.
H. N OTATIONS
Let k denote the total number of classes.
As Φ( w , x i ) is a multidimensional function for multi-class classiﬁcation, let’s denote the j th component of the output by Φ j ( w , x i ) .
Also, denote the margin for j th class corresponding to i th data point( j (cid:54) =
y i ) by ρ i,j , i.e, ρ i,j =
Φ y i ( (cid:101) w , x i ) − Φ j ( (cid:101) w , x i ) .
Margin for a data point i is deﬁned as ρ i =
min
j (cid:54) =
y i ρ i,j .
The margin for the entire network is deﬁned as ρ = min i ρ i .
Also, deﬁne a matrix M ( w ) of dimensions ( m, k ) , that is given by
[Equation]
Also, for a matrix A , vec ( A ) represents the matrix vectorized column-wise.
H. A SSUMPTIONS
The assumptions can be broadly divided into loss function/architecture based assumptions and trajectory based assumptions.
The loss functions/architecture based assumptions are shared across both gradient ﬂow and gradient descent.
Loss function/Architecture based assumptions
[Equation]
) 2 Φ( ., x ) is a C 2 function, for a ﬁxed x 3 Φ( λ w , x ) =
λ L Φ( w , x ) , for some λ > 0 and L >
Gradient ﬂow .
For gradient ﬂow, we make the following trajectory based assumptions
[Equation]
All the assumptions above are exactly the same as for exponential loss, except for (A3).
Using assumption (A1), we can say
[Equation]
Thus, we can say, for large enough t , (cid:96) i ≈ (cid:80)
j (cid:54) =
y i e − (Φ yi ( w [Equation] − Φ j ( w ( t ) , x i )) .
Thus, assumption (A3) basically states that, not just the loss vector converges in direction, but its components corresponding to various classes also converge in direction.
This is required to show that gradients converge in direction in case of multi-class classiﬁcation.
Gradient Descent .
For gradient descent, we also require the learning rate η ( t ) to not grow too fast.
[Equation]
the network
Proposition 4.
Under assumptions (A1)-(A4), lim t →∞ η ( t ) (cid:107) w u ( t ) (cid:107)∇ w u L ( w ( t )) (cid:107) =
holds for every u in the network with η ( t )
=
O ( 1 L c ) , where c < 1 .
Proof.
The gradient of the loss function is given by ∇
[Equation]
L = [Equation] [Equation] Under Assumption (A1), w ( t ) can be represented as w ( t ) =
g ( t ) (cid:101) w + r ( t ) , where lim t →∞ (cid:107) r ( t ) (cid:107) g ( t ) = 0 and r ( t ) (cid:62) (cid:101) w = 0 .
Using the equation above, we can say
[Equation]
As the order remains the same as in the proof for exponential loss, the proof follows from Appendix
B.
This proposition establishes that the Assumption (A5) is mild and holds for constant η ( t ) , that is generally used in practice.
H. E FFECT OF N ORMALISATION ON W EIGHT AND G RADIENT N ORMS
This section contains the main theorems and the difference between EWN and SWN that makes
EWN asymptotically relatively sparse as compared to SWN.
First, we will state a common proposition for both SWN and EWN.
Proposition 5.
Under assumptions (A1)-(A4) for gradient ﬂow and (A1)-(A5) for gradient descent, for both SWN and EWN, the following hold:
[Equation]
where µ >
(ii) (cid:107) (cid:101) w u (cid:107) > 0 =
⇒ (cid:101) w u = λ (cid:101) g u for some λ >(i)
lim
t →∞
[Figure]
w L [Equation] w ( t )) (cid:107)∇ w
L ( w ( t )) (cid:107) = µ (cid:80) mi =1 (cid:80) j (cid:54) =
y i (cid:102) M
[ i, j ]( ∇ w Φ y i ( (cid:101) w , x i ) − ∇ w Φ j ( (cid:101) w , x i ))
= (cid:101) g , where µ >
Proof. using Assumption (A1), w ( t ) can be represented as w ( t ) =
g ( t ) (cid:101) w + r ( t ) , where lim t [Equation] r ( t ) (cid:107) g ( t )
= 0 .
Then,
[Equation]
∇
[Equation]
Now, using ρ > 0 and Euler’s homogeneity theorem, we can say
[Equation]
Thus, (cid:107)∇ w Φ j ( w , x i ) − ∇ w Φ y i ( w , x i ) (cid:107) > 0 for all i, j .
Using these facts, Equation (39), Equation (40) and Equation (41), we can say
[Equation]
(cid:88) i = (cid:88) j (cid:54) = y i (cid:102) M [ i, j ]( ∇ w Φ y i ( (cid:101) w , x i ) − ∇ w Φ j ( (cid:101) w , x i ))
(ii) (cid:107) (cid:101) w u (cid:107) > 0 = ⇒ (cid:101) w u = λ (cid:101) g u for some λ >
Proof.
The proof follows from Appendix C.
The ﬁrst and second part state that under the given assumptions, for both SWN and EWN, gradients converge in direction and the weights that contribute to the ﬁnal direction of w , converge in opposite direction of the gradients.
Now, we provide the main theorem that distinguishes SWN and EWN.
Theorem 4.
Under assumptions (A1)-(A4) for gradient ﬂow and (A1)-(A5) for gradient descent, the following hold
(i) for EWN,
[Equation]
> 0 ,
[Equation]
(ii) for SWN, (cid:107)
[Table]
w
[Table]
>
[Table]
Proof.
The proof follows from Appendix D.
H. S PARSITY I NDUCTIVE B IAS FOR E XPONENTIAL W EIGHT N ORMALISATION
The following proposition shows why EWN is likely to converge to relatively sparse points.
Proposition 6.
Let assumptions (A1)-(A5) be satisﬁed.
Consider two nodes u and v in the network such that (cid:107) (cid:101) g v (cid:107) > 0 , (cid:107) (cid:101) g u (cid:107) > 0 , (cid:107) w u ( t ) (cid:107) → ∞ and (cid:107) w v ( t ) (cid:107) → ∞ . Let (cid:107) (cid:101) g u (cid:107) (cid:107) (cid:101) g v (cid:107) be denoted by c . Let (cid:15), δ be such that 0 < (cid:15) <
c and 0 < δ < 2 π .
Then, the following holds: There exists a time t 1 , such that for all t > t 1 both SWN and EWN trajectories have the following properties: (a) (cid:107)∇ w u L ( w ( t )) (cid:107) (cid:107)∇ w v L ( w ( t )) (cid:107) ∈ [ c − (cid:15), c
+ (cid:15) ] (b) (cid:16) w u ( t ) (cid:107) w u ( t ) (cid:107) (cid:17) (cid:62) (cid:16) [Table] [Equation] [Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
cos
[Table]
[Equation] for SWN, lim t →∞ (cid:107) w u ( t ) [Table] (cid:107) w v ( t ) (cid:107) =
c 3. for EWN, if at some time t 2 > t 1 ,
[Equation]
The above proposition shows that the limit property of the weights in Theorem 4 makes non-sparse w an unstable convergent direction for EWN, while that is not the case for SWN.
Proof.
The proof follows from Appendix E.
H. C ONVERGENCE RATES
Theorem 5.
For EWN, under Assumptions (A1)-(A5) and lim t →∞ (cid:107) r ( t +1) − r ( t ) (cid:107) g ( t +1) − g ( t ) = 0 , the following hold
(cid:107) w (cid:107) asymptotically grows at Θ (cid:16) (log( d ( t ))
1 L (cid:17) 2.
L ( w ( t )) asymptotically goes down at the rate of Θ (cid:16) 1 d ( t )(log d ( t ) [Equation] (cid:17) .
Proof.
The proof follows Appendix G.2, the only difference is in the gradient update.
Let w be represented as w = g ( t ) (cid:101) w + r ( t ) , where lim t →∞ (cid:107) r ( t ) (cid:107) g ( t ) = 0 .
Using Equation (41), we can say
[Equation]
As the order remains the same as in the proof for exponential loss, the proof follows from Appendix
G.2.
I L EMMA P ROOFS
Lemma 1.
Under assumptions (A1)-(A4) for gradient ﬂow and (A1)-(A5) for gradient descent, for both SWN and EWN, (cid:101) w (cid:62) u (cid:101) g u ≥ 0 for all nodes u in the network.
Proof.
We will show the proof just for exponential parameterization and gradient descent, but other cases can be handled similarly.
We only need to consider nodes having (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) g u (cid:107) > 0 , as for other nodes (cid:101) w (cid:62) u (cid:101) g u = 0 .
Consider a node u having (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) g u (cid:107) > 0 .
Let’s say (cid:101) w (cid:62) u (cid:101)
g u < 0 .
This means, there exists a time t 1 , such that for any t > t 1 , w u ( t ) (cid:62) ( −∇ w u L ( w ( t ))) < 0 .
Then using Equation (11), we can say that, for t > t 1 , α u ( t + 1) <
α u ( t ) .
But, this contradicts the assumption that (cid:107) w u (cid:107) → ∞ .
Thus, (cid:101) w (cid:62) u (cid:101) g u ≥ 0 .
Lemma 2.
Under assumptions (A1)-(A4) for gradient ﬂow and (A1)-(A5) for gradient descent, for both SWN and EWN, there exists atleast one node u in the network satisfying (cid:107) (cid:101) w u (cid:107) > 0 and (cid:107) (cid:101) g u (cid:107) > 0 .
Proof.
Under the assumption that ρ > 0 and using Euler’s homogeneous theorem, using Proposition 2, we can say
[Equation]
Thus, there must be atleast one node s satisfying (cid:107) (cid:101) w s (cid:107) > 0 and (cid:107) (cid:101) g s (cid:107) > 0 .
Similarly, it can be shown for cross-entropy loss as well.
Lemma 3.
Consider two unit vectors a and b satisfying a (cid:62) b ≥ 0 and a (cid:62) b < 1 .
Then, there exists a small enough (cid:15) > 0 , such that for any unit vector c satisfying c (cid:62) a ≥ cos( (cid:15) ) and any unit vector d satisfying d (cid:62) b ≥ cos( (cid:15) ) , b (cid:62) ( I − cc (cid:62) )
d ≥ (cid:15) .
Proof.
First we will try to ﬁnd bounds on b (cid:62) c and c (cid:62) d .
[Equation]
Now, using the fact that c (cid:62) a ≥ cos( (cid:15) ) and d (cid:62) b ≥ cos( (cid:15) ) , we can say (cid:107) c − a (cid:107) ≤ (cid:112) 2 − 2 cos( (cid:15) ) and (cid:107) d − b (cid:107) ≤ (cid:112) 2 − 2 cos( (cid:15) ) .
Using these bounds and the equation above, we can say
[Equation]
Using these, we can say
[Equation]
Now, we need to show that there exists an (cid:15) > 0 such that cos( (cid:15) ) − ( a (cid:62) b + (cid:112) 2 − 2 cos( (cid:15) ))
( a (cid:62) b + 2 (cid:112) 2 − 2 cos( (cid:15) )
+ (2 − 2 cos( (cid:15) )))
> (cid:15) .
At (cid:15) = 0 , LHS takes the value 1 − ( a (cid:62) b ) 2 , while RHS takes the value 0.
Thus, by continuity with respect to (cid:15) of the functions involved, we can say that there exists an (cid:15) > 0 for which the condition is satisﬁed.
Lemma 4.
Consider sequence a satisfying the following properties
a k >2.
(cid:80) ∞ k =
a k = ∞ 3.
lim k →∞ a k =
Then (cid:80) ∞
k =
[-a k (cid:113)(cid:80) k j =
-][Equation]
∞
Proof.
If (cid:80) ∞ k =
a 2 k is bounded, then the statement is obvious.
Let’s consider the case when (cid:80) ∞ k =
a 2 k diverges.
As lim k →∞ a k = 0 , therefore there must be an index k 1 , such that for k ≥ k 1 , a k ≤ (cid:15) .
Now, as a k ≤ (cid:15) , therefore a 2 k ≤ (cid:15)a k .
Now, as (cid:80) ∞ k =
a 2 k diverges, therefore, there must be an index k 2 > k 1 , such that for any k > k 2 , (cid:80) kj =
k 1 a 2 j ≥ (cid:80) k 1 − 1 j =
a 2 j .
Now, for k > k 2 , we can say[-k (cid:88) j = ka j (cid:113)(cid:80) jl =
 a 2 l ≥ 1 √ k (cid:88) j = ka j (cid:113)(cid:80) jl =-]
[Equation]
As (cid:80) ∞ k =0 a k diverges, therefore (cid:80) ∞
k = a k (cid:113)(cid:80) kj [-=
-][Equation]
0 a 2 j diverges as well.
Lemma 5.
Consider two sequences a and b satisfying the following properties
a k > 0 , (cid:80) ∞
k =0 a k = ∞ and lim k →∞ a k =
b 0 > 0 , b is increasing and b 2 k +1 ≤ b 2 k
+ ( a k b k )
Then (cid:80) ∞
k =
a k b k = ∞ .
Proof.
As we know b is increasing and b 2 k +1 ≤ b 2 k
+ ( a k b k ) 2 , we get
[Equation]
Using this, we can say
[Equation]
Now, if (cid:80) ∞ k =
a 2 k does not diverge to inﬁnity, then b remains bounded using the bound above and then its trivial to establish that (cid:80) ∞ k =
a k b k diverges.
In case, (cid:80) ∞
k =
a 2 k diverges to inﬁnity, then there must be an index k 1 such that for any k > k 1 , we can say (cid:80) k − 1 j =0 a 2 j ≥ b 40 .
So, for k > k 1 , we can say
[Equation]
Now, as we have assumed a tends to zero, so there must be an index k 2 such that for any k > k 2 , a k ≤ (cid:15) .
Also, as we have assumed (cid:80) ∞
j =0 a 2 j diverges, therefore there must be an index k 3 > k 2 , such that for k > k 3 , (cid:80) kj = k 2 a 2 j ≥ (cid:80) k 2 j =
a 2 j .
Using these things and that if a j ≤ (cid:15) , then a 2 j ≤ (cid:15)a j , we can say for k > k 3 , k
[Equation]
=
k [Equation]
Now, as (cid:80) ∞ k =0 a k diverges, thus (cid:80) ∞ k =
a k b k diverges as well.
Lemma 6.
Consider two sequences a and b satisfying the following properties
a k > 0 and (cid:80) ∞
k =0 a k = ∞ 2.
b k > 0 and (cid:80) ∞
k =
b k = ∞ 3.
(cid:80) ∞ k =
( a k − b k ) converges to a ﬁnite value 4.
lim k →∞ a k b k exists
Then lim k →∞ a k b k = 1 .
Proof.
Let’s say
lim k →∞ a k b k = c > 1 .
The other case can be handled similarly.
Choose an (cid:15) >such that c − (cid:15) > 1 .
Then, there exists an index k 1 , such that for k > k 1 , we can say
[Equation]
Using this, we can say, for k > k 1 ,
[Equation]
Summing the equation above from k 1 to ∞ and recognizing that (cid:80) ∞ k =0 b diverges, we get (cid:80) ∞
k = k 1 ( a k − b k )
= ∞ .
This contradicts.
Therefore lim k →∞ a k b k = 1 .
J I NTEGRAL F ORM OF S TOLZ -C ESARO T HEOREM
We ﬁrst state the Stolz-Cesaro Theorem.
Theorem.
(Muresan, 2015) Assume that { a } ∞ k =1 and { b } ∞ k =1 are two sequences of real numbers such that { b } ∞ k =1 is strictly monotonic and diverging.
Additionally, if lim k →∞ a k +1 − a k b k +1 − b k = L exists, then lim k →∞ a k b k exists and is equal to L .
Now, we state and prove the Integral Form of Stolz-Cesaro Theorem.
Theorem.
Consider two functions f ( t ) and g ( t ) greater than zero satisfying (cid:82) b a f ( t ) dt < ∞ and (cid:82) b a g ( t ) dt < ∞ for every ﬁnite a, b .
For any time t, its known that (cid:82) ∞ t f ( t ) dt = ∞ and (cid:82) ∞ t g ( t )
dt = ∞ .
If lim t →∞ f ( t ) g ( t ) exist and is equal to L , then lim t →∞ (cid:82) tc f ( t ) dt (cid:82)
tc g ( t ) dt exists for any c and is equal to L .
Proof.
Case 1: L = 0 or ∞ :
We will prove for L = ∞ .
The case for 0 can be handled similarly.
For any M > 0 , there must exist a time t 1 > c , such that f ( t ) g ( t ) > M , for t > t 1 .
Thus we can say for t > t 1 ,
[Equation]
Adding M (cid:82) t 1 c g ( t ) dt on both the sides, we get
[Equation]
Dividing both sides by (cid:82) t c g ( t ) dt and taking limsup t → ∞ (using also the fact that (cid:82) b a f ( t ) dt < ∞ and (cid:82) b a g ( t ) dt < ∞ for every ﬁnite a, b ), we get
[Equation]
Similarly the equation holds for liminf as well.
Thus, both liminf and limsup are greater than M for any M .
Hence lim t →∞ (cid:82) tc f ( t ) dt (cid:82) tc g ( t )
dt = ∞ .
Case2: L is
ﬁnite
In this case, there must exist some time t 1 > c , such that L − (cid:15) < f ( t ) g ( t ) <
L + (cid:15) .Thus
, we can say for t > t 1 ,
[Equation]
dt
Taking the left inequality, adding ( L − (cid:15) ) (cid:82) t 1 c g ( t ) dt on both the sides, dividing both the sides by (cid:82) t c g ( t ) dt and taking lim inf t →∞ , we get
[Equation]
Similarly, taking the right inequality, adding ( L + (cid:15) ) (cid:82) t 1 c g ( t ) dt on both the sides, dividing both the sides by (cid:82) t c g ( t ) dt and taking lim sup t →∞ , we get
[Equation]
Using the two inequalities, we get, for any (cid:15) > 0 ,
[Equation]
Thus, lim t
[Equation]
tc f
[Equation]
t
[Equation]
dt
[Equation]
tc g ( t ) dt exists and is equal to L .
K S TANDARD W EIGHT N ORMALIZATION IS NOT L OCALLY L IPSCHITZ IN ITS PARAMETERS
In this section, we will denote w by θ so as to be consistent with the notaion in Lyu & Li (2020).
SWN(in its parameters γ and v ) is also a homogeneous network.
Therefore, results from Lyu & Li (2020) should directly apply to the case of SWN as well.
However, a crucial point to be noted is that it is not even locally Lipschitz around (cid:107) v u (cid:107)
= 0 .
Therefore, the assumptions from Lyu & Li (2020) do not hold.
However, during gradient descent or gradient ﬂow, if started from a ﬁnite (cid:107) v u (cid:107) > 0 , for all u , then during the entire trajectory, (cid:107) v u (cid:107) cannot go down.
Therefore, the network is still locally Lipschitz along the trajectory it takes.
Examining the proofs from Lyu & Li (2020), its clear that the proof regarding monotonicity of margin and convergence rates are just dependent on the path that gradient descent/ﬂow takes and thus the proofs hold.
However, the result regarding the limit points of θ (cid:107) θ (cid:107) do not hold.
One of the crucial theorems the proof relies on is stated below
Theorem.
Let { x k ∈ R d : k ∈ N } be a sequence of feasible points of an optimization problem (P), { (cid:15) k > 0 : k ∈ N } and { δ k > 0 : k ∈ N } be two sequences.
x k is an ( (cid:15) k , δ k ) -KKT point for every k and (cid:15) k
→ 0 , δ k → 0 .
If x k → x as k → ∞ and MFCQ holds at x , then x is a KKT point of (P)
The above statement requires MFCQ to be satisﬁed at x , that was shown in Lyu & Li (2020) assuming local lipschitzness/smoothness at x .
However, in this case, for gradient ﬂow, as (cid:107) v u (cid:107) does not grow, while | γ u | → ∞ , therefore the convergent point of θ (cid:107) θ (cid:107) will always have the component corresponding to v u as 0.
Thus, the network is not locally lipschitz at x and the proof that MFCQ holds is violated.
Similarly, for gradient descent as well, it can’t be said that v u has a non-zero component in θ (cid:107) θ (cid:107) .
Thus, the proof does not hold.
L E XPERIMENT D ETAILS
In all the experiments, techniques for handling numerical underﬂow were used as described in Lyu & Li (2020).
However, the learning rate they used was of O ( 1 L ) , but in our case, we generally modify it to be O ( 1 L c ) , where c < 1 .
L. L IN -S EP
The learning rate used was k ( t ) L 0 .
97 , so that it speeds up at the beginning of training, but slows down as loss approaches e − 300 .
The constant k ( t ) was initialized at 0 .
01 , and was increased by a factor of 1 . 1 every time loss went down and decreased by a factor of 1 . 1 every time loss went up after a gradient step.
Its value was capped at 0 .
01 for EWN and SWN.
L. S IMPLE -T RAJ
The learning rate used was k ( t ) L 0 .
9 , so that it speeds up at the beginning of training, but slows down as loss approaches e − 50 .
The constant k ( t ) was initialized at 0 .
01 , and was increased by a factor of 1 .every time loss went down and decreased by a factor of 1 . 1 every time loss went up after a gradient step.
Its value was capped at 0 .
1 for EWN and Unnorm.
L. XOR
The learning rate used was k ( t ) L 0 .
93 , so that it speeds up at the beginning of training, but slows down as loss approaches e − 50 .
The constant k ( t ) was initialized at 0 .
01 , and was increased by a factor of 1 .every time loss went down and decreased by a factor of 1 . 1 every time loss went up after a gradient step.
Its value was capped at 0 .
01 for EWN and 0 .
1 for other cases.
L. C ONVERGENCE RATE EXPERIMENT
For all SWN, EWN and Unnorm, the learning rate was constant η = 0 .
and they were trained for 5000 steps.
All the networks were explicitly initialized to the same point in function space.
L. MNIST P RUNING E XPERIMENT
The learning rate used was k ( t ) L .
The constant k ( t ) was initialized at 0 .
01 , and was increased by a factor of 1 . 1 every time loss went down and decreased by a factor of 1 . 1 every time loss went up after an epoch.
Its value was capped at 0 .
01 for all the cases.
M R ESULTS FOR S TANDARD
[Table]
[Figure]
|| w u | | ||
w
||
|| w
|| ||
w
||
||
w
||
||
w
||
|| w
||
||
w
|| ||
w
|| ||
w 8 ||
[Figure]
10000 Steps 1.
C
o s i n e b / w w u a n d g (a)  10000 Steps 1.
C
o s i n e b / w w u a n d w u w 5 ww(b) 1.
1.8 log ||
w u
|| 189.
l o g
||
w u
|| y - x = -191.
w 5 w 6 w(c) 1.
1.9 log ||
w u
|| 237.
l o g
||
w u
|| y - x
= -239.
w 5 w 6 w(d) 1.
2.0 log || w u || 286.
l o g ||
w u
|| y - x = -287.
w 5 w 6 w(e) w u L(cid:107) − log
[Equation]
w
[Figure] [Figure]