H OW D OES A DAPTIVE O PTIMIZATION I MPACT L OCAL N EURAL N ETWORK G EOMETRY ?
Anonymous authors
Paper under double-blind review
A
BSTRACT
Adaptive optimization methods are well known to achieve superior convergence relative to vanilla gradient methods.
The traditional viewpoint in optimization, particularly in convex optimization, explains this improved performance by arguing that, unlike vanilla gradient schemes, adaptive algorithms mimic the behavior of a second-order method by adapting to the global geometry of the loss function.
We argue that in the context of neural network optimization, this traditional viewpoint is insufﬁcient.
Instead, we advocate for a local trajectory analysis.
For iterate trajectories produced by running a generic optimization algorithm OPT, we introduce R OPTmed , a statistic that is analogous to the condition number of the loss Hessian evaluated at the iterates.
Through extensive experiments, we show that adaptive methods such as Adam bias the trajectories towards regions where R Adammed is small, where one might expect faster convergence.
By contrast, vanilla gradient methods like SGD bias the trajectories towards regions where R SGDmed is comparatively large.
We complement these empirical observations with a theoretical result that provably demonstrates this phenomenon in the simpliﬁed setting of a two-layer linear network.
We view our ﬁndings as evidence for the need of a new explanation of the success of adaptive methods, one that is different than the conventional wisdom.
I NTRODUCTION
The efﬁcient minimization of a parameterized loss function is a core primitive in statistics, optimization and machine learning.
Gradient descent (GD), which iteratively updates a parameter vector with a step along the gradient of the loss function evaluated at that vector, is a simple yet canonical algorithm which has been applied to efﬁciently solve such minimization problems with enormous success.
However, in modern machine learning, and especially deep learning, one frequently encounters problems where the loss functions are high dimensional, non-convex and non-smooth.
The optimization landscape of such problems is thus extremely challenging, and in these settings gradient descent often suffers from prohibitively high iteration complexity.
To deal with these difﬁculties and improve optimization efﬁciency, practitioners in recent years have developed many variants of GD.
One prominent class of these GD variants is the family of adaptive algorithms (Duchi et al., 2011; Tieleman et al., 2012; Kingma & Ba, 2015).
At a high level, adaptive methods scale the gradient with an adpatively selected preconditioning matrix, which is constructed via a moving average of past gradients.
These methods are reminiscent of second order gradient descent, since they construct approximations to the Hessian of the loss functions, while remaining computationally feasible since they eschew full computation of the Hessian.
A vast line of empirical work has demonstrated the superiority of adaptive methods over GD to optimize deep neural networks, especially on Natural Language Processing (NLP) tasks with transformers (Vaswani et al., 2017;
Devlin et al., 2019).
From a theoretical perspective, adaptive methods are well understood in the traditional context of convex optimization.
For instance, Duchi et al.
(2011) show that when the loss function is convex, then the Adagrad algorithm yields regret guarantees that are provably as good as those obtained by using the best (diagonal) preconditioner in hindsight.
The key mechanism that underlies this improved performance, is that the loss function has some global geometric property (such as sparsity or a coordinate wise bounded Lipschitz constant), and the algorithm adapts to this global geometry by adaptively selecting learning rates for features that are more informative.
However, in non-convex optimization, and deep learning in particular, it is highly unclear whether this simple characterization is sufﬁcient to explain the superiority of adaptive methods over GD.
Indeed, for large scale neural networks
[Figure]
global guarantees on the geometric properties of the loss are typically vacuous.
For instance, for a 20-layer feedforward neural network, if we scale up the weights in each layer by a factor of 1 . 5 , then the global Lipschitz constant of the network is scaled up by a factor of at least e 10 .
Hence it only makes sense to study convergence by looking at the local geometry of the loss along the trajectory of the optimization
algorithm (Arora et al., 2018)
[Figure] [Figure]
Moreover, the interaction between an optimization algorithm and neural network geometry is highly complex — recent work has shown that geometric characteristics of iterates encountered during optimization is highly dependent on the choice of optimization algorithm and associated hyperparameters (Lewkowycz et al., 2020; Cohen et al., 2021).
For instance, Cohen et al.
(2021) demonstrate that while training neural networks with GD, the maximum eigenvalue of the Hessian evaluated at the GD iterates ﬁrst increases and then plateaus at a level 2/(step size).
The viewpoint from convex optimization, where a loss function has some (potentially) non-uniform but ﬁxed underlying geometry that we must adapt to, is thus insufﬁcient for neural networks, since the choice of optimization algorithm can actually interact with and inﬂuence the observed geometry signiﬁcantly .
To provide another example of this interactive phenomenon, we consider the following experiment.
On the same network training loss function f , we run stochastic gradient descent with momentum (SGD+M) and Adam to obtain two different trajectories.
We select an iterate x Adam from the Adam trajectory and an iterate x SGD from the SGD trajectory, such that f ( x Adam ) =
f ( x SGD ) .
We then run
SGD+M twice, once from x Adam and once from x SGD .
If the underlying geometry of the loss function f was truly ﬁxed, then we would not expect a signiﬁcant difference in the performance of running
SGD+M from either of the two iterates.
However, as shown in Figure 1(left), running SGD+M from x Adam achieves lower loss than that from x SGD , suggesting that Adam may bias the trajectory towards a region which is more favorable for rapid training.
This motivates the following question.
How does adaptive optimization impact the observed geometry of a neural network loss function,
relative to SGD (with momentum)?
The remainder of this paper is dedicated to answering the above question.
To this end, for each iterate in a trajectory produced by running an optimization algorithm OPT, where the Hessian of the t th iterate is given by H ( t ) ∈ R d × d , we deﬁne the second order statistic R OPTmed ( t ) in the following fashion.
For the t th iterate in the trajectory, let R OPTmed ( t ) be the ratio of maximum of the absolute entries of the diagonal of H ( t ) , to the median of the absolute entries of the diagonal of H ( t ) .
Concretely, we deﬁne
R OPTmed
[Equation]
This statistic thus measures the uniformity of the diagonal of Hessian, where a smaller value of R OPTmed ( t ) implies that the Hessian has a more uniform diagonal.
It can also be viewed as a stable 1 variant of the condition number.
Instead of eigenvalues, we choose diagonal entries because adaptive methods used in practice are coordinate-wise , which can be viewed as the diagonal scaling approaches.
In Appendix A.9 we discuss this intuition in detail and compare R OPTmed ( t ) with singular valuebased metrics.
As a supplementary result, in Appendix E, we demonstrate that the loss Hessian approaches diagonal during training for Adam and SGD+M. There has been prior theoretical work on overparameterized neural networks showing that a smaller condition number of Hessian, Neural
Tangent Kernel (Jacot et al., 2018) etc. could yield to faster convergence rate for (S)GD (Liu et al., 2022).
As for (diagonal) adaptive methods ( e.g. Adagrad), they were original designed to adapt to the nonuniform diagonal geometry.
Intuitively, a smaller R OPTmed ( t ) , which implies more uniform diagonal geometry, could lead to faster convergence.
Armed with this statistic, we make the following contributions:
• On a wide variety of neural network transformer architectures and language modeling datasets, we conduct experiments to compare how R Adammed ( t ) and R SGDMmed ( t ) evolve over time, when Adam and SGD+M are run from the same initialization and with their optimal (initial) learning rates respectively.
In each case, we demonstrate that the Adam trajectory attains R Adammed ( t ) values that are signiﬁcantly smaller than the R SGDMmed ( t ) values found by SGD+M.
We show a simple example of this phenomenon in Figure 1(right).
This suggests that relative to SGD+M, Adam biases the optimization trajectory to a region where the Hessian diagonal is more uniform.
We call this phenomenon the uniformity of diagonal geometry for adaptive methods.
As an aside, we observe that larger improvements in optimization performance of Adam over SGD+M are correlated with larger gaps between R Adammed ( t ) and R SGDMmed ( t ) .
This suggests that a region where the Hessian diagonal is more uniform is also a region that is more amenable to rapid optimization.
• We complement our empirical results with a theoretical analysis of this phenomenon in the simpliﬁed
setting of large batch Adam and SGD+M, on a two-layer linear network with d -dimensional input and hidden layer, and one dimensional output.
We show that for a wide range of t , R Adammed ( t ) = 1 ± o (1) but R SGDMmed ( t ) = Ω(log d ) .
Our proof reveals that Adam induces the weight matrices to have low rank whose leading singular vectors have certain type of uniformity (see Section 6 for discussion), a fact that we also observe empirically in large scale neural networks, suggesting that this may be a mechanism by which adaptive methods bias trajectories to have uniformity of diagonal geometry.
R ELATED WORK
Existing analyses of adaptive methods.
The vast majority of prior theoretical work on adaptive methods has focused on the blackbox setting (Duchi et al., 2011; Kingma & Ba, 2015;
Chen et al., 2020; Reddi et al., 2018; Ward et al., 2020; D ´ efossez et al., 2020; Ene et al., 2021).
These works make minimal assumptions about the structure of the loss function, beyond (possibly) some global properties such as convexity or smoothness.
These global properties (governed by parameters such as the smoothness parameter) are assumed to hold over the entire domain.
Hence this style of analysis is worst case, since the resulting convergence bounds depend on polynomially on these global parameters.
However, as we show in Section 3.1, in neural networks these parameters are prohibitively large.
This worst case analysis is hence unlikely to explain the success of adaptive methods on neural networks.
By contrast, our focus is on analyzing the local trajectory that is induced by running the optimization method.
Existing analyses of (S)GD on neural networks.
There is an extensive literature on the analysis of
GD/SGD in the non-blackbox setting, e.g. overparameterized neural networks, (Du et al., 2018; Ji & Telgarsky, 2020; Allen-Zhu et al., 2019a;b; Arora et al., 2019a; Liu et al., 2022).
However, it is unclear how to translate these analyses of GD/SGD, to an analysis that explains the gap between
GD/SGD and adaptive methods.
Inﬂuence of algorithms on the loss geometry.
In many simple convex settings, e.g. linear or logistic regression and the Neural Tangent Kernel (Jacot et al., 2018), the loss geometry is usually ﬁxed and not inﬂuenced by learning algorithms.
However, in neural networks the interaction between algorithms and loss landscapes is more complicated.
Lewkowycz et al.
ﬁnd a so-called catapult effect of initial learning rate on the training trajectory of SGD and related loss curvature.
Cohen et al.
(2021) demonstrate that while training neural networks with GD, the maximum eigenvalue of the Hessian evaluated at the GD iterates ﬁrst increases and then plateaus at a level that is inversely proportional to the step size.
However, Cohen et al.
(2021) leave open the problem of whether similar interactive phenomena occur in algorithms that are not GD, including adaptive methods.
O VERVIEW OF RESULTS AND SETUP 3. I SSUES OF PRIOR ANALYSES ON ADAPTIVE METHODS
As is mentioned in Section 2, existing work on adaptive algorithms has mainly focused on blackbox analysis assuming some global worst-case parameters.
However, these global bounds can be extremely bad in complicated deep learning models, as is discussed in Section 1.
To see this, we initialized a transformer model 3 with default initialization in Pytorch but chose a large gain 4 , and computed the smoothness parameter (denoted as l ) and the condition number (denoted as κ ) of loss Hessian on one layer.
We observed that setting the gain as a large constant ( e.g. 800) results in extremely large l and κ ( l ≥ 10 7 and κ ≥ 10 10 ), which makes the convergence rates in prior black-box analysis vacuous.
The failure of global worst-case analysis implies that we need to focus on the local trajectory of algorithms.
However, it is unclear that when two optimization algorithms are used, they will have the same geometry in local trajectory.
In particular, although in theory, adaptive algorithms can yield to a convergence rate with better dependency on certain local geometry of the function comparing to SGD (with momentum), it could still be the case that the local geometry along the trajectory of adaptive algorithm can be much worse than that of SGD (with momentum).
That motivates us to study the local geometry, especially that obtained by adaptive methods comparing to SGD (with momentum) in the paper.
Motivated by the diagonal scaling of Adagrad and Adam for neural network training, we ask the follow main question in our paper:
How does the local diagonal geometry (diagonal of the loss Hessian) along the local trajectory of adaptive algorithms compare to that of SGD (with momentum)?
3. O VERVIEW OF THE EXPERIMENTS
As is discussed in Section 1, we consider R OPTmed ( t ) deﬁned in eq.
(1) as a measurement of the uniformity of the diagonal of the loss Hessian.
We conduct experiments on different NLP tasks to examine
R OPTmed ( t ) , as in language models, adaptive methods have shown signiﬁcantly faster convergence than
SGD (with momentum).
The details of these experiments will be shown in Section 4.
To explore potential different patterns of different layers, we do the computation layer by layer.
On a wide variety of transformer architectures and language modeling datasets from the same initialization, we observe that:
When we train the neural network using Adam, the uniformity of diagonal geometry, measured by R OPTmed ( t ) is smaller than that when we train using SGD+M from the same initialization, except for ﬁrst several layers.
Table 1 shows a typical example of R Adammed ( t ) compared to R SGDMmed ( t ) on a sentence classiﬁcation task using BERT-small (Turc et al., 2019;
Bhargava et al., 2021)
(See Section 4.1 for details).
We repeated the experiments for 12 times starting from the same initialization.
Table 1 shows the averaged R Adammed ( t ) and R SGDMmed ( t ) in some randomly selected layers (except for the ﬁrst several).
We also report the averaged R SGDM med ( t ) R Adammed ( t ) and their standard deviations in the brackets.
5 Figure 2 shows the corresponding training losses of one in these 12 experiments.
[Table]
To understand this phenomenon in a more principled point of view
[Table]
we also provide a formal proof of the statement in a simpliﬁed setting: large batch Adam and SGD+M on a 2-layer linear network.
Although simple, the choice of 2-layer linear network to understand learning dynamics is common in prior works ( e.g. (Tian et al., 2021)).
Section 3.3 below describes the theoretical setup.
3. S ETUP OF THE THEORETICAL ANALYSIS
Notation Let [ d ] = { 1 , 2 , ..., d } .
We use (cid:107) · (cid:107) 2 to denote the l 2 norm of a vector, and (cid:107) · (cid:107) F to denote the Frobenius norm of a matrix.
Let (cid:104)· , ·(cid:105) be the Euclidean inner product between vectors or matrices.
Let N ( µ, σ 2 ) be the one-dimensional Gaussian distribution with mean µ and variance σ 2 .
For a scalar (vector, matrix)
A which evolves over time, we use A ( t ) to denote its value at time t .
Let there be m data points.
The data matrix is X ∈ R d x × m and the label matrix is Y ∈ R d y × m .
We assume that the input dataset is whitened, i.e. Λ xx :
=
1 m XX T ∈ R d x × d x is an identity matrix.
The parameters of a 2-layer linear network are given by W :=
( W 2 , W 1 ) .
Assume W
i ∈ R d i ×
d
i − 1 for i = 1 , 2 .
We have d 2 =
d y , d 0 = d x .
We consider the square loss L ( W ) :
= 12 m (cid:107) W 2 W 1 X − Y (cid:107) 2 F .
Denote A :=
1 m Y X T ∈ R d y × d x .
Arora et al.
(2019b) show that with whitened dataset,
[Equation]
where c does not depend on W .
We consider the following model with small Gaussian initialization.
Assumption 1 (Setup).
The input covariance Λ xx := 1 m XX T ∈ R d x × d x is an identity matrix.
The input and hidden layers are both of dimension d , i.e. d 1 = d 0 = d .
Without loss of generality, we can assume that A is a row vector (i.e. d 2 = 1 ) whose coordinates are positive 6 and Θ(1) in terms of d .
Assumption 2 (Gaussian Initialization).
∀ i, j : w (0)2 i ∼ N (0 , 1 d 2 α ) , W (0)1 [ i, j ] ∼ N (0 , 1 d 4 α ) are independently initialized with sufﬁciently large α > 0 .
Denote ˜ A and ˜Λ xx as the batch versions of A and Λ xx .
We make the following large-batch assumption.
We emphasize that large batches are commonly used in NLP tasks ( e.g. (Brown et al., 2020)).
Assumption 3 (Large Batch).
For the randomly selected batches, assume E [ ˜ A ]
= A , E [˜Λ xx ]
= Λ xx .
∀
[Equation] [Equation]
Denote ˜ g ( t ) as the batch gradient at time t .
The update rules of SGD+M and Adam are given by
[Equation]
where η is the learning rate, β, β 1 , β 2 are momentum parameters, and ξ is for numerical stability.
All operations on vectors are element-wise.
Here and throughout, the notation f ( x ) =
O ( g ( x ))
(resp.
f ( x ) =
Ω( g ( x )) , f ( x ) =
Θ( g ( x )) ) means that there exist constants C 1 , C 2 >
such that f ( x ) ≤ C 2 g ( x ) (resp.
f ( x ) ≥ C 1 g ( x ) , C 1 g ( x ) ≤ f ( x ) ≤ C 2 g ( x ) ).
We will also use the notation with ∼ , i.e. ˜ O ( · ) , ˜Ω( · ) , ˜Θ( · ) to hide factors that are logarithmic in d .
In our theoretical analysis, “with high probability”, or “w.h.p.”
for short, means that with probability at least 1 − 1 poly ( d ) .
T HE UNIFORMITY OF DIAGONAL GEOMETRY
As is mentioned in Section 3.2, we computed R OPTmed ( t ) deﬁned in eq.
(1) on different language models.
In this section, we present the results of SGD+M and Adam on different architectures and datasets.
In Appendix A, we present the results of other adaptive algorithms.
During training we started from the same initial weights and used the same learning rate schedule (constant or decreasing) for SGD+M and Adam.
We tuned and chose the best (initial) learning rate of SGD+M. The (initial) learning rate of Adam was set as a value under which Adam converged faster than SGD+M with its best learning rate.
The concrete values will be stated in later parts of this section.
We used large batch sizes to make the training procedure stable.
When computing Hessian, we also used large batch sizes.
Due to the extremely large dimension, we did the computation on some uniformly selected coordinates, more precisely, 200 coordinates per layer.
4. E XPERIMENTS ON REAL DATASETS
Sentence classiﬁcation task on BERT-small We ﬁne-tuned BERT-small (Turc et al., 2019; Bhargava et al., 2021) on the IMDB dataset (Maas et al., 2011): the task is to classify whether movie reviews are positive or negative.
The momentum parameter β in SGD was set as 0.9.
The two momentum parameters ( β 1 , β 2 ) of Adam were set as (0.9, 0.999).
We trained the model using linearly decreasing learning rates for 10 epochs (2500 iterations).
The initial learning rates of SGD+M and Adam were 0.001 and 5e-5, respectively.
As mentioned in Section 3.2, Figure 2 and Table 1 show the training losses and the comparison between R Adammed ( t ) and R SGDMmed ( t ) , respectively.
Translation task We trained a Seq2Seq network that uses Transformer to solve a machine translation task on Multi30k (Elliott et al., 2016)(CC BY-NC-SA 4.0): this task is to train a German to English translation model.
The momentum parameter β in SGD was set as 0.9.
The two momentum parameters ( β 1 , β 2 ) of Adam were set as (0.9, 0.98).
We trained the model using constant learning rates (0.03 for SGD+M and 1e-4 for Adam) for 60 epochs (1800 iterations).
The experiments were repeated for 8 times starting from the same initialization.
Figure 3(left) shows the training losses for one among them.
Table 2a shows the averaged R Adammed ( t ) , R SGDMmed ( t ) and R SGDMmed ( t ) R Adammed ( t ) (with standard deviation in the brackets) in some randomly selected layers.
[Figure]
[Figure]
4. E XPERIMENTS ON RANDOM DATASETS
We used the same model and momentum parameters as in the translation task described in Section 4.
but generated random integers as targets.
Similar to the setting on real targets, the model was trained
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
11.7(1.48) using constant learning rates (0.015 for SGD+M and 5e-5 for Adam) for 60 epochs (1800 iterations), and we repeated the experiments for 8 times starting from the same initialization.
Figure 3(right) shows the training losses for one among them.
Table 2b shows the averaged R Adammed ( t ) , R SGDMmed ( t ) and
R SGDMmed ( t ) R Adam med ( t ) (with standard deviation in the brackets) of the same 10 layers as in Table 2a.
4. S UMMARIZATION OF THE EMPIRICAL RESULTS AND DISCUSSION
Overall, through extensive experiments on language models, we demonstrate that starting from the same initialization, the R OPTmed ( t ) values found by Adam are smaller than those found by
SGD+M, except for the ﬁrst several layers.
This suggests that Adam is biased towards a region with more uniform diagonal Hessian than SGD+M.
In Appendix A.10 we also validate this observation on the in-distribution test data.
Positive correlation between uniformity of diagonal Hessian and fast convergence.
We observe that on random dataset, SGD+M plateaus after about 400 steps and thus converges much slower when compared to Adam than on real dataset (see Figure 3).
On the other hand, the gaps of R SGDMmed ( t ) and R Adammed ( t ) are more signiﬁcant on random data than on real data (see Table 2b) as well.
In
Appendix A.4, we conduct another experiment where we switch from SGD to Adam in the middle and compare it with the model trained by Adam from the beginning.
The observation is that both the loss gap and the gap of R OPTmed ( t ) are gradually closed after switching (see Figure 7 and Table 8).
Hence we ﬁnd a positive correlation between fast convergence and the uniformity of diagonal of loss Hessian, suggesting that a region with more uniform diagonal of Hessian is also a region that is more amenable to fast optimization.
In Appendix A we study other adaptive algorithms (Adagrad,
RMSprop and AMSGrad) and get similar observation : all these adaptive methods converge faster than SGD or SGD+M and also bias the trajectory to regions with smaller R OPTmed ( t ) , suggesting that the uniformity of diagonal Hessian might be a universal mechanism (partially) explaining the faster optimization of adaptive algorithms than SGD (with momentum).
More discussions on the trajectory difference.
Considering the fact that our comparison between
R Adammed ( t ) and R SGDMmed ( t ) is conditioned on the same iteration when SGD+M has larger training loss than Adam, there is a potential alternative explanation of the Hessian diagonal uniformity.
That is, the global minimum has uniform Hessian, and Adam simply converges faster to it than SGD+M.
To rule out this possibility, in Appendix A.3 we add a comparison of our measurements R Adammed ( t ) and R SGDMmed ( t (cid:48) ) , where t, t (cid:48) are picked such that t th Adam iterate and t (cid:48)
th SGD+M iterate have the same training loss .
The results (in Table 7) show that R Adammed ( t ) <
R SGDMmed ( t (cid:48) ) for most layers, thus demonstrating that the trajectories of Adam and SGD+M are truly different and that the difference is because Adam biases the local geometry (as opposed to faster convergence).
Adding regularization.
People in practice usually add weight decay (equivalent to l 2 regularization) to encourage better generalization ability.
In Appendix A.7 we compare SGD+M and Adam when both using small weight decay values (0.001).
The results in Figure 13a and Table 9 suggest that in this case, the positive correlation between R OPTmed ( t ) and convergence speed still holds: Adam converges faster than SGD+M and in most of the layers except for the ﬁrst several, R Adammed ( t ) values are smaller than R SGDMmed ( t ) .
This reveals the robustness of our observation under weak regularization.
However, under large weight decay parameters, we observed cases where Adam still converged faster but R Adammed ( t ) values were larger rather than smaller.
In the case of strong regularization, the adaptivity of Adam requires further exploration and we hope to ﬁnd new mechanisms in the future.
Image tasks.
Although in this paper we focus on language models where Adam shows signiﬁcant fast convergence, we also add supplementary results in Appendix A.8 on image tasks where SGD+M performs better.
On a residual network trained on CIFAR-10, we observed that Adam did not converge faster than SGD+M (see Figure 13b) and in the meantime, R Adammed ( t ) values were no longer smaller than R SGDMmed ( t ) during training (see Table 10).
This reveals the connection between the local diagonal geometry and the convergence speed from another perspective.
That is, when the diagonal of Hessian of Adam is not more uniform than SGD+M, its convergence speed is not better, either.
T HEORETICAL ANALYSIS
In Section 4, we empirically demonstrate the uniformity of diagonal geometry.
In this section, we theoretically analyze this property for large batch Adam and SGD+M on a two-layer linear network with 1-dimensional output.
Since the weights and Hessians in different layers may have different magnitudes, we compute the
R OPTmed ( t ) layer by layer.
We denote R SGDMmed ,k ( t ) (resp.
R Adammed ,k ( t ) ) as the R OPTmed ( t ) found by SGD+M (resp.
Adam) w.r.t.
W k at time t
where k = 1 , 2 .
Theorem 1.
Under Assumption 1, 2 and 3, consider the weights (cid:110) W ( t ) SGD (cid:111)
t ≥ 0 (resp.
(cid:110) W ( t )
Adam (cid:111) t ≥ 0 ) obtained by SGD+M (resp.
Adam) deﬁned in (3) .
For any p > 0 , pick 0 < (cid:15) < 1 d p , η ≤
O (cid:0) (cid:15) d 7 α/ 4+4 (cid:1) and α ≥ 4( p + 2) .
Suppose σ ≤ η 3 / 2 d α/ 2+1 , then there exists T SGD , 1 , T SGD , 2 such that w.h.p., ¯ L (cid:16) W ( T SGD , 1 ) SGD (cid:17) =
Θ( d ) , ¯ L (cid:16) W ( T SGD , 2 ) SGD (cid:17) ≤ ˜ O (cid:0) 1 d p (cid:1) , and ∀ t ∈
[ T SGD , 1 , T SGD , 2 ] :
R SGDMmed ,k ( t ) = Ω(log d ) , k = 1 , 2 .
For any p > 0 , pick η ≤ O (cid:0) 1 d 3 α (cid:1) , ξ ≤ (cid:112) η d 3 α − 1 , α ≥ p +43 and β 2 = β 21 .
Suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 ,
Then ∃ T Adam , 1 , T Adam , 2 such that w.h.p., ¯ L (cid:16) W ( T Adam , 1 ) Adam (cid:17) =
Θ( d ) , ¯ L (cid:16) W ( T Adam , 2 ) Adam (cid:17)
≤ ˜ O (cid:0) 1 d p (cid:1) , and ∀ t ∈ [ T Adam , 1 , T Adam , 2 ] :
R Adammed ,k ( t ) = 1 ± ˜ O (cid:18)
η 14 + d α 2 −(cid:19) , k
= 1 , 2 .
An immediate corollary of this theorem below gives the difference between iterates of Adam and
SGD+M that have the same loss.
Corollary 1.
Under the setup in Theorem 1, w.h.p., for any t ∈ [ T SGD , 1 , T SGD , 2 ] and t (cid:48) ∈
[ T Adam , 1 , T Adam , 2 ] such that ¯ L (cid:16) W ( t ) SGD (cid:17) =
¯ L (cid:16) W ( t (cid:48) )
Adam (cid:17) ∈ (cid:104) ˜Ω (cid:0) 1 d p (cid:1) , Θ( d ) (cid:105) , we have
R SGDMmed ,k ( t ) = Ω(log d ) ,
R Adammed ,k ( t (cid:48) )
= 1 ± ˜ O (cid:18)
η 14 + d α 2 −(cid:19) , k
= 1 , 2 .
Theorem 1 and Corollary 1 tell us that during a long training period when the loss decreases from Θ( d ) to ˜ O (cid:0) 1 d p (cid:1) , the diagonal of loss Hessian for Adam keeps nice uniformity in the sense that for each layer, its diagonal elements have roughly the same value, i.e. R Adammed ,k ( t )
= 1 ± o (1) , k
= 1 , 2 .
On the other hand, the diagonal of loss Hessian for SGD+M is less uniform.
Appendix B gives a proof sketch of Theorem 1.
The detailed proof can be found in Appendix C and D.
T HE LOW RANK STRUCTURE OF WEIGHT MATRICES AND UNIFORMITY OF LEADING SINGULAR VECTORS
The proof sketch in Appendix B highlights one crucial intuition of Theorem 1: After T SGD , 1 (resp.
T Adam , 1 ) steps, W 1 of SGD+M (resp.
Adam) becomes an approximately rank-1 matrix.
Consider the left singular vector u :=
[ u 1 , u 2 , ..., u d ] T which corresponds to the leading singular value σ 1 .
We can show that the distribution of u 21 , u 22 , ..., u 2 d for Adam is more uniform than that of SGD+M.
This property, we call the uniformity of the leading singular vector , is related to the uniformity of the diagonal of loss Hessian, see Appendix F for more details.
Similar low rank bias after training has been studied in prior works ( e.g. (Gunasekar et al., 2017;
Li et al., 2018; Chou et al., 2020)).
For more complicated models, we want to check whether the weight matrices also have low rank structures and if so, whether we can still observe the uniformity of leading singular vectors .
More formally, consider the weight matrix in some layer W ∈ R m × n , we want to check
(A) Whether W ∈ R m × n is approximately a rank k matrix with k (cid:28) min { m, n } , and (B) If (A) is true, then consider the top k singular values σ 1 , ..., σ k and corresponding left singular vectors u 1 , u 2 , ...
u k .
Deﬁne a new vector ˜ u
= (cid:80) ki
=1 σ 2 i u i (cid:12) u i :=
[˜ u 1 , ˜ u 2 , ..., ˜ u
d ] T and compute R u := max i ˜
u i median ˜
u
i , which is a generalized version of max i u 2 i median u 2 i in the rank 1 case.
We want to see whether R u obtained by Adam is smaller than that of SGD+M.
After reviewing the weight matrices we got in different settings, we observed that (A) and (B) hold for many layers in those models.
For example, on the translation task mentioned in Section 4.1, we found 12 layers which had approximately low rank structures and for 10 of them, R u values (deﬁned in (B)) obtained by Adam were smaller than those found by SGD+M. Figure 4 shows the result on one typical layer.
Results of more layers can be found in Appendix A.5.
Remarks 1.
The deﬁnition of R u is based on the connection between diagonal of loss Hessian and weight matrices.
Appendix F shows that for a 2-layer linear network, R OPTmed , 2 ( t ) = max i (cid:107) W ( t )
[ i, :] (cid:107)median (cid:107) W ( t )
[ i, :] (cid:107) 22 .
When W 1 ∈ R m × n is approximately rank k , i.e. W 1 ≈ (cid:80) ki =
σ i u i v Ti , denote u i =
[ u i 1 , u i 2 , ..., u im ] T and v i =
[ v i 1 , v i 2 , ..., v in ] T , we have that for the j -th row, (cid:107) W
[Equation] [Equation]
By deﬁning ˜ u
=
[˜ u 1 , ˜ u 2 , ..., ˜ u
d ] T := (cid:80) ki
=1 σ 2 i u i (cid:12) u i , we have that (cid:107) W 1 [ j, :] (cid:107) 22 ≈ ˜
u
j .
Although in multi-layer nonlinear neural networks, the connection between diagonal of loss Hessian and the weight matrices is more complicated and R OPTmed , 2 ( t ) may depend on the product of many weight matrices rather than one single matrix, we still believe that this deﬁnition of R u is a reasonable ratio to consider.
We may also want to consider the right singular vectors v 1 , v 2 , ... v k and corresponding ˜ v =
[˜ v 1 , ˜ v 2 , ..., ˜ v d ] T := (cid:80) ki
=1 σ 2 i v i (cid:12) v i and compute R v :=
max i ˜ v i median ˜ v i for Adam and SGD+M.
However, on this translation task, among the 12 layers which were approximately low rank, for only 6 of them,
R v of Adam were smaller, i.e. we did not observe uniformity of the leading right singular vector for Adam.
Results of R v can be found in Appendix A.5.
One possible reason is that for a weight matrix, its right singular vectors are closer to the input data than left singular vectors and more easily inﬂuenced by the data, therefore may not show uniformity.
C ONCLUSION AND FUTURE WORK
We demonstrate that adaptive optimization methods bias the training trajectory towards a region where the diagonal of loss Hessian is more uniform, through extensive experiments on language models and theoretical analysis in a simpliﬁed setting of two-layer linear networks.
Although our ﬁndings may not directly lead to an improved algorithm for practical use, they provide a new way of thinking when designing new algorithms: in contrast with the traditional view which tries to design a method that performs better in the bad loss geometry, our ﬁndings suggest that we can design algorithms which implicitly avoid regions with bad geometry.
There are a lot of future directions along this line.
For example, our theoretical results on the two-layer linear networks may be able to generalize to multi-layer networks.
In fact, people conjecture that the key-value-query structure in language models can be approximated by a three-layer linear network.
Hence the generalization to multi-layer networks might provide more connection to real deep models and could be an interesting and challenging future direction.
Moreover, it is also possible to relax our large-batch assumption (Assumption 3) and prove similar results in the general stochastic setting.
R EPRODUCIBILITY S TATEMENT
The training details (e g. hyperparameters) of experiments are speciﬁed in Section 4.
The source code is provided in the supplemental material.
For the theoretical results, Section 3.3 states the full set of assumptions.
Section C and Section D in the appendix provide complete proofs
A M ORE EXPERIMENTS OF THE UNIFORMITY OF DIAGONAL GEOMETRY A. V ANILLA SGD VS . A DAGRAD
In this section, we present the R OPTmed ( t ) values deﬁned in eq.
(1) obtained by vanilla SGD and Adagrad on a language modeling task 10 .
The task is to assign a probability for the likelihood of a given word (or a sequence of words) to follow a sequence of words.
We trained a transformer model to solve this problem on both Wikitext-2 (Merity et al., 2017)(CC BY-SA 3.0) and random dataset (generating random integers as targets).
This model has roughly 8 layers (not counting normalization and dropout layers)
The setup is the same as in Section 3.2.
We used the same learning rate schedule (constant or decreasing) for SGD and Adagrad.
We tuned and chose the best (initial) learning rate of SGD.
The (initial) learning rate of Adagrad was set as a value under which Adagrad converged faster than
SGD with its best (initial) learning rate.
We used large batch sizes to make the training procedure more stable.
When computing Hessian, we also used large batch sizes.
Due to the extremely large dimension, we did the computation on some uniformly selected coordinates, more precisely,coordinates per layer.
We tried different initialization (normal and uniform) by using different gains of the Pytorch initialization schedule.
A.1. E XPERIMENTS ON REAL DATASET
Figure 5a shows the training losses on real dataset (wikitext-2).
Table 3 (resp.
Table 4) shows the
R OPTmed ( t ) for Adagrad and SGD under uniform (resp. normal) initialization with different gains.
[Figure]
T r a i n i ng l o ss
SGDAdagrad
[Figure]
[Table] [Table]
A.1. E XPERIMENTS ON RANDOM DATASET
Figure 5b shows the training losses on random dataset and Table 5 shows the R OPTmed ( t ) in different layers.
[Table]
A. RMS PROP AND AMSG RAD
In this section
[Table]
we present the results of RMSprop and AMSGrad and compare them with SGD+M.
The experiments were conducted on the translation task described in Section 4.1.
The learning rates we used were 2.5e-5 for RMSprop, 0.0005 for AMSGrad and 0.03 for SGD+M.
Both RMSprop and
SGD+M used momentum parameter 0.9.
The two momentum parameters ( β 1 , β 2 ) of AMSGrad were (0 . 9 , 0 . 98) .
Figure 6 shows the training losses and Table 6 shows the corresponding R OPT med ( t ) .
[Figure]
[Table]
A. C OMPARISON CONDITIONED ON THE SAME LOSS
In this section
[Table]
we compare R SGDMmed ( t ) and R Adammed ( t ) conditioned on the same training loss.
More precisely
[Table]
we make comparison of R Adammed ( t ) and R SGDMmed ( t (cid:48) ) , where t, t (cid:48) are picked such that t th Adam iterate and t (cid:48)
th SGD+M iterate have the same training loss.
The details of the tasks are described in in Section 4.1.
Table 7 shows the results of R Adammed ( t ) and R SGDMmed ( t (cid:48) ) in some layers.
[Table]
R SGDMmed ( t (cid:48) )
R Adammed ( t ) R SGDMmed ( t (cid:48) )
R Adammed ( t )4.
[Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
A. E XPERIMENTS OF SWITCHING FROM SGD TO A DAM
In this section we describe another learning schedule: the “Adam after SGD” schedule, where we switch from SGD to Adam in the middle to see whether the loss and R OPTmed ( t ) can catch up with the model trained by Adam from the very beginning.
Again, we used the same model as in the translation task in Section 4.1.
In this section, we did not add momentum term to SGD in order to get a larger gap between SGD and Adam than the case using momentum.
We want to see whether this larger gap can be closed after switching to Adam in the middle.
As is shown in Figure 7 and Table 8, both the loss gap and the gap of R OPTmed ( t ) were closed after a period of training after switching algorithms, which provides evidence of the connection between convergence speed and uniformity of diagonal of loss Hessian.
A. T HE LOW RANK STRUCTURE
In this section, we present more results for the experiments in Section 6.
We examined the weights of the model trained for the translation task in Section 4.1.
Among roughly 30 layers, we observed that for 12 layers, at least the weight matrices obtained by Adam after training have approximately low rank structures.
Figure 8 shows the examples of layers with or without the low rank structure.
[Figure]
We then studied the uniformity of leading singular vectors of these 12 layers, i.e. computed R u and
R v deﬁned in (B) and the second remark of Section 6.
The observation is that for 10 out of theselayers, R u values of Adam were smaller those of SGD, which implies the uniformity of leading left singular vectors of Adam.
However, we did not observe signiﬁcant uniformity for Adam in terms of leading right singular vectors ( R v ).
The second remark of Section 6 discusses possible reasons.
Figure 9 shows how R u and R v changed over time in some layers.
[Figure]
A. H OW THE ( ADAPTIVE ) GRADIENT ALIGNS WITH DIAGONAL OF LOSS H ESSIAN
In this section we present the uniformity of diagonal geometry of adaptive methods from another perspective.
Denote H ii as the ( i, i ) -th element of the loss Hessian H and g i as the i -th element of the gradient.
It is conjectured that when | H ii | is large, the corresponding | g i | is usually large as well.
For adaptive methods, we can regard the update per step as the learning rate times the “adaptive gradient”.
Let’s use g adapt ,i to represent the i -th component of the adaptive gradient.
Through experiments on language models, we found that | g adapt ,i | for different i are quite uniform and do not align with | H ii | as the true gradient | g i | does.
In the experiments, we ﬁrst sorted | H ii | in the ascent order: | H i 1 ,i 1 | ≤ | H i 2 ,i 2 | ≤ ... ≤ | H i
d ,i d | (suppose H ∈ R d × d ), and then plotted the corresponding |
g i k | and | g adapt ,i k | for k ∈ [ d ] .
A.6. SGD VS . A DAGRAD
Here we compare SGD and Adagrad on the language modeling task on wikitext-2 described in
Section A.1.
We observed that the ﬁgures of all layers are quite similar so we select one layer as an example, as is shown in Figure 10.
[Figure]
| g adap t, i |
A.6. SGD WITH MOMENTUM VS . A DAM
Here we compare Adam and SGD+M on the tasks described in Section 4.1.
Again, we select one layer as an example for each task.
Figure 11 shows the results on the sentence classiﬁcation task and
Figure 12 shows the results on the translation task.
A. A DDING REGULARIZATION AND OTHER TRICKS
In this section, we add weight decay to both Adam and SGD+M on the translation task described in Section 4.
The momentum parameter β in SGD was set as 0.9.
The two momentum parameters ( β 1 , β 2 ) of Adam were set as (0.9, 0.98).
For both algorithms, we set the weight decay parameter as 0.001.
We trained the model using constant learning rates for 60 epochs (1800 iterations).
We tuned and chose the best learning rate 0.03 for SGD+M.
The learning rate of Adam was set as 0.0001, under which Adam converged faster than SGD+M with its best learning rate 0.03.
Figure 13a shows the
[Figure] [Figure]
training losses and Table 9 shows the values of R Adammed ( t ) , R SGDMmed ( t ) and R SGDM med ( t ) R Adam med ( t ) in some randomly selected layers.
A. R ESULTS ON IMAGE TASKS
We trained a ResNet 11 on CIFAR-10 dataset and compared the convergence speed and R OPTmed ( t ) of
SGD+M and Adam.
The momentum parameter β in SGD was set as 0.9.
The two momentum parameters ( β 1 , β 2 ) of Adam were set as (0.9, 0.98).
The model was trained using constant learning
[Figure]
[Table]
rates for 41 epochs (2050 iterations).
We tuned and chose the best learning rates for both algorithms: 0.5 for SGD+M and 0.005 for Adam.
Figure 13b shows the training losses and Table 10 shows the values of R Adammed ( t ) , R SGDMmed ( t ) and R SGDMmed ( t ) R Adammed ( t ) .
[Table]
A. C OMPARISON BETWEEN R OPT MED ( t ) AND SINGULAR VALUE - BASED METRICS
In Section 4, through extensive experiments on language models, we demonstrate that when we train the neural network using Adam, the uniformity of diagonal geometry, measured by R OPTmed ( t ) is smaller than that when we train using SGD+M from the same initialization, for most of the layers.
We are aware that people also usually consider Hessian singular values instead of diagonal entries to measure the loss geometry.
Hence in this section we make a comparison between our diagonal-based metric and singular value-based metrics.
First, we believe that our metric has a natural connection to the mechanism that underlies adaptive methods.
Adaptive methods in practice choose coordinate-wise adaptive learning rates.
From a high-level perspective, this procedure can be viewed as adapting to the loss smoothness with respect to each coordinate.
The smoothness of certain coordinate is measured by the second derivative with respect to this coordinate and therefore corresponds to the diagonal entries of loss Hessian.
Our metric, which measures these diagonal entries, is thus fundamentally intertwined with the mechanism that underlies adaptive methods.
Next, we empirically demonstrate that our metric R OPTmed ( t ) is a reasonable proxy of singular valuebased metrics.
Deﬁne a singular value-based metric S OPTmed ( t ) :
= max { σ i ( t ) } di =1 median { σ i ( t ) } di =1 as an analogy of our diagonal-based metric R OPTmed ( t ) , where { σ i ( t ) } di =1 denotes the singular values of loss Hessian
H ( t ) ∈
R
d × d at the t th iterate.
We compare S OPTmed ( t ) along the trajectories of Adam and SGD+M in the translation task described in Section 4.1.
Table 11 suggests that if measured by singular values,
Adam is also biased to a region with smaller S OPTmed ( t ) than SGD+M, similar to the observation for
R OPTmed ( t ) .
This is expected because in Appendix E, we demonstrate that the loss Hessian approaches diagonal during training.
The fact that our diagonal-based metric and singular value-based metric give the same result also reveals the robustness of our observation to the choice of metric, demonstrating that there does exist some geometry bias of Adam towards more uniform regions even when measured by different metrics.
Finally, there is strong reason why our metric is often easier to compute empirically and analyze theoretically than singular value-based metrics such as S OPTmed ( t ) .
From the
empirical
computation perspective,
suppose
the loss
Hessian is
d ×
d
Then
computing
its singular values,
in
general, requires
computing the
whole
matrix
with d
elements
However, our metric only requires
computing
the d diagonal entries.
From
the theoretical analysis perspective
, in
Appendix F
we
show
that the diagonal of loss Hessian in linear networks
can
be connected to weight matrices by simple formulas
These straightforward formulas simplify the analysis and allow us to connect our metric to the low-rank structure of weight matrices and the uniformity of their leading singular vectors (see Section 6 for more discussions).
However, all these nice connections fail to hold for singular value-based metrics.
The formulas of singular values are very complicated even in linear networks, making it almost impossible to theoretically analyze any singular value-based metrics.
[Table]
A. T HE UNIFORMITY OF DIAGONAL GEOMETRY ON IN - DISTRIBUTION TEST DATA
In this section we compare R Adammed ( t ) and R SGDMmed ( t ) on the in-distribution test data.
The task is the translation task described in Section 4.1.
Table 12 validates that on in-distribution test data, Adam is also biased to a region with smaller R OPTmed ( t ) than SGD+M, similar to what happens on the training data shown in Table 2a.
This is expected because of the same distribution.
One other thing we want to emphasize is that, in real language tasks, the dataset is typically very large and the model see each training example only once.
Hence the training behavior usually implies similar in-distribution test behavior.
[Table]
B P ROOF SKETCH OF T HEOREM
Now we give a proof sketch of Theorem 1, which contains three major steps.
The detailed proof can be found in Appendix F, C and D.
First we relate the diagonal of Hessian to weight matrices W 1 , W 2 .
Under Assumption 1, denote
W 1 [ i, :] as the i -th row of W 1 and W 2 :=
[ w 2 i , w 22 , ..., w 2 d ] .
Since the input dataset is whitened, we can show that
[Equation]
Next, due to the one-dimensional output, we can prove that W 1 converges to an approximately rankmatrix.
More precisely, we have
[Equation]
where c ( t ) is a scalar, u ( t ) , v ( t ) , R ( t )∈ R d and R ( t )
∈ R
d × d .Denote the i -th coordinate of u ( t ) , v ( t ) , R ( t ) 2 as u ( t ) i , v ( t ) i , R ( t ) 2 i , respectively.
Denote the ( i, j ) -th element of R ( t ) 1 as R ( t ) 1 [ i, j ] .
We have that ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12)
[Equation]
( t ) (cid:12)(cid:12)(cid:12) u ( t ) i (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12) R ( {+t ) 
+}
[ i, j ] (cid:12)(cid:12)(cid:12) (cid:28) (cid:12)(cid:12)(cid:12) u ( t ) i v ( t ) i (cid:12)(cid:12)(cid:12) .
Using the rank 1 structure, we can further simplify R OPTmed , 1 ( t ) and R OPTmed , 2 ( t ) by
[Equation]
The ﬁnal step is the detailed analysis of u ( t ) .
For SGD+M, we can prove that u ( t ) ≈ C ( t )
[ X 1 , X 2 , ..., X d ] T where C ( t ) ∈
R and X i , i ∈
[ d ] are i.i.d.
Gaussian variables.
Then we have with high probability, max i
[Table]
= Ω(log d ) .
For Adam, we can prove that ∀ i ∈
[ d ] : u ( t ) i ∈ {± 1 } , which gives us max i
[Table]
= 1 .
Substituting into eq.
(4) completes the proof.
C A NALYSIS OF SGD+M
Note that A = 1 m Y X T , Λ xx :
= 1 m XX T .
Denote g ( t ) k :=
∇ W k L ( W ( t ) ) , k = 1 , 2 .
We have that
[Equation]
Let ˜ A ( t ) , ˜Λ ( t ) xx and ˜ g ( t ) k , k = 1 , 2 be the corresponding batch versions at time t .
Let E ( t ) :=
W ( t ) 2 W ( t ) 1 − A , and use E ( t ) i , A i and (cid:16) W ( t ) 2 W ( t ) 1 (cid:17) i to represent the i -th coordinates of E ( t ) , A and
W ( t ) 2 W ( t ) 1 , respectively.
By eq.
(2), the update rules of W 1 and W 2 for SGD+M are given by:
[Equation]
where
[Equation]
Based on the magnitude of W 2 and W 1 , we can intuitively divide the training procedure into 2 phases.
First phase : the ﬁrst several iterations when W 1 and W 2 are “small” so that W 2 W 1 − A ≈ − A . 2.
Second phase : later iterations when W 2 W 1 cannot be ignored.
More formally, the boundary between the ﬁrst and second phase is deﬁned below.
Deﬁnition
(End of the ﬁrst phase).
The end of the ﬁrst phase (denoted as T 1 ) is deﬁned as
T
[Equation]
( t ) 2 i (cid:12) (cid:12)(cid:12) ≥ 1 d α 2 or (cid:12) (cid:12)(cid:12) W ( t ) 1 [ i, j ] (cid:12) (cid:12)(cid:12) ≥ 1 d α 2 (cid:111) .
By Assumption 2 and the assumption that ∀ j ∈ [ d ] : A j > 0 , A j = Θ(1) , at the beginning, w.h.p., ∀ j ∈ [ d ] : ( W 2 W 1 ) j − A j < 0 .
During the training, each ( W 2 W 1 ) j increases and approaches A j .
We hope that by choosing a small learning rate, when ( W 2 W 1 ) j overshoots for some coordinate j , i.e.
( W 2 W 1 ) j > A j , it will be close to convergence.
To analyze this overshooting issue more carefully, let’s ﬁrst deﬁne the following “almost overshooting time”.
Deﬁnition 2 (Almost overshooting time).
For (cid:15) > 0 , denote (cid:15) 0 :=
[Equation]
Deﬁne
[Equation]
Deﬁnition 3 (Convergence time).
For (cid:15) > 0 , we deﬁne the “convergence time”
T 3 :=
inf (cid:110) t ≥ 0 : (cid:13)(cid:13) E ( t ) (cid:13)(cid:13) 2 2 ≤ (cid:15) (cid:111) .
We can ﬁrst show that after the ﬁrst phase, i.e. when t = T 1 , W 1 will become an approximately rank-1 matrix, as described in the following lemma.
Lemma 1.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 d α/ 2+1 .
By picking η ≤ O (cid:0) 1 d α (cid:1) , we have that when t = T 1 , ¯ L (cid:0) W ( T 1 ) (cid:1) =
Θ( d ) , and that
[Equation]
where c ( T 1 ) ∈
R , u ( T 1 ) , v ( T 1 ) , R ( T 1 )∈ R d and R ( T 1 )
∈ R d × d .
Denote the i -th coordinate of u ( T 1 ) , v ( T 1 ) , R ( T 1 )as u ( T 1 ) i , v ( T 1 ) i ,
R
( T 1 ) 2 i , respectively, and the ( i, j ) -th element of R ( T 1 )as
R ( T 1 )[ i, j ] .
Then w.h.p., ∀ 1 ≤ i, j ≤ d : (cid:12)(cid:12)(cid:12) R ( T 1 )
[ i, j ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) u
( T 1 ) i v ( T 1 ) j (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:18) d 14 α −(cid:19) , (cid:12)(cid:12)(cid:12) R ( T 1 ) 2 i (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) c
( T 1 ) u ( T 1 ) i (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:18)
d
[Equation]
The following lemma tells us that this approximate rank-1 structure is preserved when T 1 ≤ t ≤ min { T 2 , T 3 } .
Lemma 2.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 d α/ 2+1 .
By picking η ≤ O (cid:16)
[Equation]
d
[Equation] [Equation]
(cid:17) , we have that w.h.p.
for T 1 ≤ t ≤ min { T 2 , T 3 } ,
[Equation]
where
[Equation]
and (cid:15) 0 is deﬁned in Deﬁnition 2.
Moreover, when t = min { T 2 , T 3 } , ¯ L (cid:0) W ( t ) (cid:1) =
O ( (cid:15) 0 d ) .
The following lemma gives us a more detailed description of u ( T 1 ) .
Lemma 3.
The u ( T 1 ) in Lemma 1 and 2 can be written as u ( T 1 ) =
X + Y
where X i , i ∈ [ d ] are i.i.d Gaussian random variables and that w.h.p.
∀ i ∈
[ d ] : | Y i | | X i | ≤ ˜ O (cid:16) d 14 α −(cid:17) .
Now we are ready to prove the SGD+M part of Theorem 1.
C. P ROOF OF THE SGD+M PART OF T HEOREM
Deﬁne T SGD , 1 = T 1 , T SGD , 2 = min { T 2 , T 3 } .
By picking η ≤ O (cid:16) (cid:15) d 7 α 4 + (cid:17) , we can apply Lemmaand 2 to conclude that ¯ L (cid:0) W ( T SGD , 1 ) (cid:1) =
Θ( d ) and ¯ L (cid:0) W ( T SGD , 2 ) (cid:1) =
O ( (cid:15) 0 d ) .
For any p > 0 , by picking 0 < (cid:15) < 1 d p and α ≥ 4( p + 2) , we have ¯ L (cid:0) W ( T SGD , 2 ) (cid:1) =
O ( (cid:15)
d ) ≤ ˜ O (cid:0) 1 d p (cid:1) .
Moreover, when t ∈ [ T SGD , 1 , T SGD , 2 ] , the conditions in Lemma 30 are satisﬁed with δ =
˜ O
( (cid:15) 0 ) .
Then we can apply Lemma 30 and get that
[Equation]
median
[Equation]
(cid:17)
[Equation]
By Lemma 3, u ( T 1 ) =
X + Y where w.h.p.
∀ i ∈
[ d ] : | Y i | | X i | ≤ ˜ O (cid:16) d 14 α −(cid:17) .
This fact yields ∀ i ∈
[ d ] :
[Equation]
Here X i , i ∈
[ d ] are i.i.d Gaussian random variables by Lemma 3.
To prove the concentration of median X 2 i , we borrow the Proposition 12 in Chapter 2.3 of (Lerasle, 2019).
By setting K = N = d in this proposition, we have
[Equation]
Denote σ 2 as the variance of X i , i ∈
[ d ] .
Then E [ X 2 i ] =
σ 2 and Var ( X 2 i ) =
2 σ 4 .
Hence
[Equation]
That means with high probability, median X 2 i ≤ Cσ 2 for some C > 0 .
By Lemma 34 in Appendix G, we know that w.h.p.
[Equation]
, which gives us w.h.p.
[Equation]
Hence we have proved that R SGDMmed , 1 ( t ) , R SGDMmed , 2 ( t ) ≥ Ω(log d ) .
C.
P ROOF OF L EMMA
In the ﬁrst phase, W 2 W 1 is “small”, and we write the update equations in the following way
[Equation]
τ =
[Equation]
τ =
[Equation]
A
[Equation]
=
[Equation]
(5) where r ( t ) 1 = − β t +1 W ( t ) T
A +
[Equation]
t (cid:88) τ
= β t − τ (cid:16) W ( τ ) T− W ( t ) T(cid:17)
A − (1 − β ) t (cid:88) τ = β t
[Equation]
τ W (
[Equation]
[Equation]
W ( τ ) 2 W ( τ )
[Equation]
[Equation]
t (cid:88) τ
= β t − τ
Dg ( τ ) 1 .
Similarly, we have
[Equation]
where
[Equation]
The following lemma gives us an explicit formula of W ( t ) 2 .
Lemma 4.
Let λ 1 < λ 2 be the two roots of the quadratic equation x 2 − 2 x + 1 −
[Equation]
Pick η < 1 − β (cid:107) A (cid:107) 2 , then we have that
[Equation]
where C 1 = − W (
[Equation]
)2 − λ 2 W
[Equation]
2 λ
[Equation]
−
[Equation] [Equation]
r ( t ) 5 will be speciﬁed in the proof.
We can prove that in the ﬁrst phase, r ( t ) 5 is “small”.
More speciﬁcally, denote its i -th coordinate as r ( t ) 5 i , and the i -th coordinate of C 2 as C 2 i .
Then the following lemmas tell us that ∀ i ∈
[ d ] , (cid:12)(cid:12)(cid:12) r ( t ) 5 i (cid:12)(cid:12)(cid:12) ≤
O (cid:0) 1 d p ( α ) (cid:1) , where w.h.p.
O (cid:0) 1 d p ( α ) (cid:1) (cid:28) min i ∈
[ d ] | C 2 i | .
We ﬁrst have the following bounds of (cid:12) (cid:12) (cid:12) r ( t ) 1 i (cid:12) (cid:12) (cid:12) , (cid:12) (cid:12) (cid:12) r ( t ) 2 i (cid:12) (cid:12) (cid:12) and (cid:12) (cid:12) (cid:12) r ( t ) 5 i (cid:12) (cid:12) (cid:12) for i ∈ [ d ] .
Lemma 5.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 d α/ 2+1 and pick η ≤ O (cid:0) 1 d α (cid:1) .
We have w.h.p.
for all t ≤ T 1 , ∀ i ∈ [ d ] : (cid:12)(cid:12)(cid:12) r ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) d 32 α −(cid:17) , (cid:12)(cid:12)(cid:12) r ( t ) 2 i (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) d 32 α −(cid:17) .
Lemma 6.
Under conditions of Lemma 5, we have that w.h.p.
for all t ≤ T 1 , ∀ i ∈ [ d ] : (cid:12)(cid:12)(cid:12) r ( t )
i (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) d 32 α −(cid:17) .
Next we prove upper and lower bounds of | C 1 i | and | C 2 i | for i ∈ [ d ] .
Lemma 7.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 d α/ 2+1 .
Pick η < 1 − β (cid:107) A (cid:107) 2 , we have that i)
w.h.p.
, ∀ i ∈
[ d ] : | C 1 i | ≤ ˜ O (cid:0) 1 d α (cid:1) , | C 2 i | ≤ ˜ O (cid:0)
d α (cid:1)
; ii) C 2 can be written as C 2 := 12 ( C 3 + C 4 ) where C 3 i , i ∈ [ d ] are i.i.d Gaussian random variables and that w.h.p.
∀ i ∈
[ d ] : | C
i
|
|
C i | ≤ ˜ O (cid:16) d 14 α −(cid:17) ;
iii) w.h.p.
, ∀ i ∈
[ d ] , | C 1 i | ≥ ˜Ω (cid:16)
[Equation]
Now we are ready to prove Lemma 1.
Lemma 4 tells us that
[Equation]
where λ 1 = 1 − η 1 − β (cid:107) A (cid:107) 2 and λ 2 = 1 + η 1 − β (cid:107) A (cid:107) 2 .
Under the conditions of Theorem 1 and pick η ≤ O (cid:0) 1 d α (cid:1) , by Lemma 6 and 7, we know that w.h.p.
∀ t
[Equation]
d
[Equation]
We ﬁrst prove that (cid:12)(cid:12)(cid:12) w ( t ) 2 i (cid:12)(cid:12)(cid:12) reaches 1 d α/ 2 for some coordinate i before (cid:12)(cid:12)(cid:12) W ( t ) 1 [ k, j ] (cid:12)(cid:12)(cid:12) for ∀ k, j ∈ [ d ] .
To see this, ﬁrst note that
W
[Equation]
t )
[Equation]
= W
[Equation]
t
[Equation]
1)+ η 1 − β W
( t − 1) T
A + η 1 − β r
( t − 1)= W (0) + η 1 − β t − 1 (cid:88) τ =
W ( τ ) T
A + η 1 − β t − 1 (cid:88) τ
= r ( τ )=
W (0) + η 1 − β (cid:32)
Ct − 1 (cid:88) τ
= λ τ
+ Ct − 1 (cid:88) τ
= λ τ 2 + t − 1 (cid:88) τ
= λ τ 2 r ( τ )(cid:33) T
A + η 1 − β t − 1 (cid:88) τ
= r ( τ )=
W (0) + η 1 − β t − 1 (cid:88) τ
= r ( τ )
+ η 1 − β (cid:32)
Ct − 1 (cid:88) τ
= λ τ 1 + t − 1 (cid:88) τ
= λ τ 2 r ( τ )(cid:33) T
A + η 1 − β t − 1 (cid:88) τ
= λ τ 2 C T 2 A := W (0)
+ η 1 − β t − 1 (cid:88)
τ
= r
[Equation]
(cid:33)
[Equation] [Equation]
+ u ( t ) v ( t ) T , where v ( t ) T =
η 1 − β A and u (
[Equation]
Moreover, we have that
[Equation]
For t ≤ T 1 , by eq.
(7), we get that w.h.p.,
[Equation]
= (cid:12)(cid:12)(cid:12) r ( t ) 5 i (cid:12)(cid:12)(cid:12) | C 2 i | ≤ ˜ O (cid:18)
d
α −(cid:19) .
For t ≤ T 1 , by Lemma 5, ∀ 1 ≤ i, j ≤ d : (cid:12)(cid:12)(cid:12) r ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) d 32 α −(cid:17) .
Then we have that w.h.p.
(cid:12)(cid:12)(cid:12) η 1 − β (cid:80) t − 1 τ =
r ( τ )
[ i, j ] (cid:12)(cid:12)(cid:12) (cid:12) (cid:12)(cid:12) u ( t ) i v ( t ) j (cid:12) (cid:12)(cid:12)
= (cid:12)(cid:12)(cid:12)(cid:80) t − 1 τ =
r
( τ )
[ i, j ] (cid:12)(cid:12)(cid:12) (cid:12) (cid:12)(cid:12) (cid:80) t − 1 τ =0 λ
τ 2 C 2 i
A j (cid:12) (cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12)(cid:80) t − 1 τ =
r
( τ )
[ i, j ] (cid:12)(cid:12)(cid:12) (cid:12) (cid:12)(cid:12) (cid:80) t −
1 τ =
C 2 i
A j (cid:12) (cid:12)(cid:12) ≤ (cid:80) t −
[Table]
τ =
˜ O (cid:16) d 32 α −(cid:17)
[Equation]
Here we used ∀ i ∈
[ d ] : A i = Θ(1) by Assumption 1.
Since λ 1 = 1 − η 1 − β (cid:107) A (cid:107) 2 , we have that | C 1 i λ t 1 | ≤ | C 1 i | ≤ ˜ O (cid:0)
d α (cid:1) and that
[Equation]
Using the Gaussian tail bound and union bound, we have w.h.p.
∀ 1 ≤ i, j ≤ d : (cid:12)(cid:12)(cid:12) W (0)
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:0) 1 d 2 α (cid:1) .
Combining the above bounds together yields that for t ≤ T 1 and ∀ i, j ∈ [ d ] ,
[Equation] [Equation]
[Table]
t
[Table] [Table]
we
[Table]
∀ j ∈ [ d
[Table]
[Equation]
Together with eq.
(9) gives us that
[Equation]
reaches 1 d
[Equation]
/ 2 for some
i ∈
[ d ] before (cid:12)(cid:12)(cid:12) W ( t
[Equation]
[ k, j ] (cid:12)(cid:12)(cid:12) for ∀ k, j ∈ [ d ] , i.e.
T 1 =
[Equation]
Further, we know that at time T 1 , (cid:12)(cid:12)(cid:12) c ( T 1 ) u ( T 1 )
[Equation]
0 (cid:12)(cid:12)(cid:12) = | C
[Equation]
i 0 | λ T 1 2 = Θ (cid:0) 1 d α/ 2 (cid:1) for some i 0 ∈ [ d ] , which means w.h.p.
[Equation]
This is the length of the ﬁrst phase.
As for c ( T 1 ) u ( T 1 ) i and u ( T 1 ) i v ( T 1 ) j for other coordinates, we have that w.h.p.
∀ 1 ≤ i, j ≤ d , (cid:12)(cid:12)(cid:12) u ( T 1 ) i v ( T 1 )
j (cid:12)(cid:12)(cid:12)
[Equation]
(cid:88)
[Equation]
Here in ( i ) we used λ 2 = 1 + η 1 − β (cid:107) A (cid:107) 2 .
Then we have at time T 1 , ∀ i, j ∈
[ d ] , (cid:12)(cid:12)(cid:12) R ( T
[Equation]
11 [ i,j ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12) (cid:12) u ( T 1) i v ( T 1) j (cid:12)(cid:12) (cid:12) ≤ ˜ O
cid:16)d 14 α (cid:17) and that (cid:12)(cid:12)(cid:12) R ( T 1) 21 ,i (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) c ( T 1) u ( T 1) i (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) 1 d 14 α (cid:17) .
Together with eq.
(9), we have the following weight structure:
[Equation]
where w.h.p., ∀ 1 ≤ i, j ≤ d :
[Equation] [Equation]
Finally, we consider the loss.
Since ∀ j ∈ [ d ] : (cid:16) W ( T 1 )
W ( T 1 )(cid:17) j − A j = − Θ(1) , we know that ¯ L (cid:0) W ( T 1 ) (cid:1) =
Θ( d ) .
C. P ROOF OF L EMMA
Eq.
(8) tells us that u ( T 1 ) =
(cid:80) T 1 − 1 τ =
λ τ 2 C T 2 .
Lemma 7 tells us that C 2 can be written as C 2 :
= 12 ( C 3 + C 4 ) where C 3 i , i ∈
[ d ] are i.i.d Gaussian random variables and that w.h.p.
∀ i ∈
[ d ] : | C 4 i | | C 3 i | ≤ ˜ O (cid:16) d 14 α −(cid:17) .
Combining these two facts together ﬁnishes the proof.
C. P ROOF OF L EMMA
Replacing t by t − 1 in eq.
(6), we get
[Equation]
Eq.
(6)-(11) and substituting eq.
(5) yield
[Equation]
where r 3 ( t ) :
=
[Equation]
For the equation x 2 − 2 x + 1 − η 2 (1 − β ) 2 (cid:107) A (cid:107) 22 = 0 , the roots are λ 1 = 1 − η 1 − β (cid:107) A (cid:107) 2 and λ 2 = 1 + η 1 − β (cid:107) A (cid:107) 2 .
We have that
[Equation]
We further have
[Equation]
where r ( t ) 5 = (cid:80) tτ
[Equation] [Equation]
− τ
[Equation]
) 4 , C 1 = − W
(1)2 − λ 2 W (0)2 λ 2 − λand C 2 = W
(1)2 − λ 1 W (0)2 λ 2 − λ.
C. P ROOF OF L EMMA
Write r ( t )= − β t +1 W ( t ) T
A + q ( t )+ q ( t )+ q ( t )where q ( t )=
(1 − β ) (cid:80) tτ
=
β t − τ (cid:16) W ( τ ) T− W ( t ) T(cid:17)
A , q ( t )=
− (1 − β ) (cid:80) tτ
=
β t − τ W ( τ ) T
W ( τ ) 2 W ( τ )and q ( t ) 14 = − (1 − β ) (cid:80) tτ
=
β t − τ Dg ( τ ) 1 .
And write r ( t ) 2 = − β t +1 AW ( t ) T+ q ( t ) 22 + q ( t ) 23 + q ( t ) 24 , where q ( t )
= (1 − β ) (cid:80) tτ
=
β t − τ A (cid:16) W ( τ ) T− W ( t ) T(cid:17) , q ( t )
= − (1 − β ) (cid:80) tτ
=
β t − τ W ( τ ) 2 W ( τ ) 1 W ( τ )
Tand q ( t ) 24 = − (1 − β ) (cid:80) tτ
=
β t − τ Dg ( τ ) 2 .
Let’s ﬁrst try to bound (cid:12)(cid:12)(cid:12) q ( t ) 12 [ i, j ] (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12) q ( t ) 22 ,i (cid:12)(cid:12)(cid:12) .
For any τ ≤ T 1 , we have that ∀ i ∈
[ d ] : (cid:12)(cid:12)(cid:12)(cid:16) W ( τ ) 2 W ( τ ) 1 (cid:17) i (cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) d (cid:88)
j = w ( τ ) 2 j W ( τ )
[ j, i ] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ d (cid:88)
j = (cid:12)(cid:12)(cid:12) w ( τ ) 2 j (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) W ( τ ) 1 [ j, i ] (cid:12)(cid:12)(cid:12) ≤ d (cid:88) i =
d α = d
α − 1 , and thus ∀ i ∈ [ d ] : (cid:12)(cid:12)(cid:12) E ( τ ) i (cid:12)(cid:12)(cid:12) =
O (1) .
Then we have for all i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) W ( τ +1)
[ i, j ] − W ( τ ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) ≤ η τ (cid:88) k
= β τ −
[Table]
[Equation]
[Table]
That gives us ∀ i, j ∈ [ d ] ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
t ) 22 ,i (cid:12)(cid:12)(cid:12) ≤ (1 − β ) t (cid:88) τ
= β t − τ d (cid:88)
j = (cid:12)(cid:12)(cid:12)
A j (cid:16) W ( τ ) 1 [ i, j ] − W ( t )
[ i, j ] (cid:17)(cid:12)(cid:12)(cid:12) ≤ η (1 − β ) t (cid:88) τ =
O (cid:18) β t − τ ( t − τ ) d α/ 2 −(cid:19) =
O (cid:16) η d α/ 2 −(cid:17) .
Then we bound (cid:12)(cid:12)(cid:12) q ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12) q ( t ) 23 ,i (cid:12)(cid:12)(cid:12) .
We have for ∀ i, j ∈ [ d ] , (cid:12) (cid:12) (cid:12) q ( t )
[ i, j ] (cid:12) (cid:12) (cid:12) ≤ (1 − β ) t (cid:88)
τ
= β t − τ (cid:12) (cid:12) (cid:12)(cid:12) w ( τ ) 2 i (cid:16) W ( τ ) 2 W ( τ ) 1 (cid:17) j (cid:12) (cid:12) (cid:12)(cid:12) ≤ (1 − β ) t (cid:88)
τ
= β t − τd α 2 · d α −= O (cid:18) d 32 α −(cid:19) , (cid:12)(cid:12)(cid:12) q ( t ) 23 ,i (cid:12)(cid:12)(cid:12) ≤ (1 − β ) t (cid:88) τ
= β t − τ d (cid:88)
j = (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) W ( t ) 2 W ( t ) 1 (cid:17) j W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12)(cid:12) ≤ (1 − β ) t (cid:88)
τ
= β t − τ d (cid:88) i =
d α − 1+ α 2 =
O (cid:18) d 32 α −(cid:19) .
Finally we use Lemma 31 to bound (cid:12) (cid:12)(cid:12) q ( t ) 14 [ i, j ] (cid:12)(cid:12)(cid:12) and (cid:12) (cid:12)(cid:12) q ( t ) 24 ,i (cid:12) (cid:12)(cid:12) .
For t ≤ T 1 , the M ( t ) 1 , M ( t ) 2 in Lemmaare upper bounded by 1 d
α 2 .
In the theorem we consider the training period before T SGD , 2 so the time T in Lemma 31 is set as T SGD , 2 .
In the following sections, we will prove that T SGD , 2 ≤
O (cid:18)
d α log( √ d/(cid:15) ) η (cid:19) .
Then by Lemma 31, we have with probability at least 1 − 1 d , for ∀ t ≤ T 1 and
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
By picking σ ≤ η 3 / 2 d α/ 2+
[Equation]
we have w.h.p.
for ∀ t ≤ T 1 and ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) Dg ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ η ˜ O (cid:16)
d α 2 (cid:17) and (cid:12)(cid:12) (cid:12) Dg ( t )
i (cid:12)(cid:12) (cid:12) ≤ η ˜ O (cid:16) 1 d α 2 − 1 (cid:17) , which yields (cid:12)(cid:12)(cid:12) q ( t
[Equation]
14 [ i
[Equation] [Equation]
Combining all the above bounds and substituting η ≤ O (cid:0) 1 d α (cid:1) gives us for ∀ t ≤ T 1 and ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) r ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) ≤ β t
[Equation]
1 (cid:12)(cid:12)(cid:12) w (
[Equation] [Equation]
O
[Equation] [Equation]
[Equation]
d
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
d
[Equation]
(cid:19) .
For t ≤ T 1 , we have ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) w ( t )
i
A j (cid:12)(cid:12)(cid:12) ≤ O (cid:0) 1 d α/ 2 (cid:1) and (cid:12)(cid:12)(cid:12)(cid:80) dj
=1 A j W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) ≤ O (cid:0) 1 d α/ 2 − 1 (cid:1) , which gives us (cid:12)(cid:12)(cid:12) r ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ O (cid:0) 1 d α/ 2 (cid:1) and (cid:12)(cid:12)(cid:12) r ( t ) 2 i (cid:12)(cid:12)(cid:12) ≤ O (cid:0) 1 d α/ 2 − 1 (cid:1) .
Substituting into eq.
(5) and eq.
(6) yields that for t ≤ T 1 and ∀ i, j ∈ [ d ] ,
[Equation]
Hence for t ≤ min (cid:110)
α log d log(1 /β ) , T 1 (cid:111) , we have ∀ i, j ∈ [ d ] ,
[Equation]
Then we know that T 1 > α log d log(1 /β ) and also get tighter bounds of (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) w ( t )
i (cid:12)(cid:12)(cid:12) for t ≤ α log d log(1 /β ) .
Now we use these new bounds to analyze (cid:12)(cid:12)(cid:12) r ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12) r ( t ) 2 i (cid:12)(cid:12)(cid:12) again.
When t ≤ α log d log(1 /β ) , we have for all i, j ∈ [ d ] , β t +1 (cid:12)(cid:12)
(cid:12) w ( t )
i
A j (cid:12)(cid:12) (cid:12) ≤ (cid:12)(cid:12) (cid:12) w ( t )
i A j (cid:12)(cid:12) (cid:12) ≤ ˜ O (cid:16) d 3 α 2 −(cid:17) and β t +1 (cid:12) (cid:12)(cid:12)(cid:80) dj
=1 A j W ( t )
[ i, j ] (cid:12) (cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12)(cid:80) dj
=1 A j W ( t ) 1 [ i, j ] (cid:12) (cid:12)(cid:12) ≤ ˜ O (cid:16) d 3 α 2 −(cid:17) .
When α log d log(1 /β )
<
t ≤ T 1 , we have β t +1 ≤ 1 d α , suggesting that ∀
i, j ∈ [ d ] , β t +1 (cid:12)(cid:12)(cid:12) w ( t )
i A j (cid:12)(cid:12)(cid:12) ≤ 1 d α ˜ O (cid:16)
d α 2 (cid:17) ≤ ˜ O (cid:16) 1 d 3 α 2 (cid:17) and β t +1 (cid:12)(cid:12)(cid:12)(cid:80) dj
=1 A j W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) ≤ 1 d α ˜ O (cid:16)
d α 2 − 1 (cid:17) ≤ ˜ O (cid:16) d 3 α 2 −(cid:17) .
Substituting into (12) completes the proof.
C. P ROOF OF L EMMA
Based on the bound in Lemma 5, we have
[Equation]
(cid:17)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ η(1 − β )d (cid:88)
j = (cid:12)(cid:12)(cid:12)
[Equation]
j
[Equation]
t
[Equation] [Equation]
j
[Equation]
+ η
[Equation]
η
[Equation]
η
[Equation]
˜
[Equation]
(cid:18)
d
[Equation]
(cid:19)
[Equation] [Equation]
(cid:18)
[Equation]
= η ˜ O (cid:18) d 32 α −
[Equation]
(cid:19) .
Since λ 1 = 1 − η 1 − β (cid:107) A (cid:107) 2 , λ 2 = 1 + η 1 − β (cid:107) A (cid:107) 2 , and note that (cid:107) A (cid:107) 2 = Θ (cid:16) √ d (cid:17) , we have that
[Equation]
r ( t )
i
[Equation]
ROOF OF
EMMA
For the equation x 2 − 2 x + 1 − η 2 (1 − β ) 2 (cid:107) A (cid:107) 22 = 0 , the roots are λ 1 = 1 − η 1 − β (cid:107) A (cid:107) 2 and λ 2 = 1 + η 1 − β (cid:107) A (cid:107) 2 , which gives us
[Equation]
where ˜ r (0)2 = − W (0)2 W (0)1 W (0) T− Dg (0)2 .
Note that this is slightly different from the deﬁnition of r (0)2 in eq.
Now let’s bound the i -th coordinate of ˜ r
In Section C.5 we have shown that w.h.p.
for ∀ t ≤ T 1 and ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) Dg ( t ) 2 i (cid:12)(cid:12)(cid:12) ≤ η ˜ O (cid:16) 1 d α 2 − 1 (cid:17) =
˜ O (cid:16) d 3 α 2 −(cid:17) , which also applies to t = 0 .
Using the Gaussian tail bound and union bound, w.p. at least 1 − δ , for ever 1 ≤ i, j ≤ d , we have that
[Equation]
≤ (cid:114)d 4 α log 2 dδ .
Then we have that w.p. at least 1 − δ , ∀ 1 ≤ i, j ≤ d : , (cid:12)(cid:12)(cid:12)(cid:16) W (0)2 W (0)1 (cid:17) i (cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) d (cid:88)
j = w (0)2 j W (0)1 [ j, i ] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Table]
d (cid:88)
j = (cid:12)(cid:12)(cid:12) w (0)2 j (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) W (0)1 [ j, i ] (cid:12)(cid:12)(cid:12) ≤ d (cid:88)
i
=
(cid:114)d 2 α log 2 d δ (cid:114)d 4 α log 2 dδ ≤ d 3 α − 1 log 2 dδ ,
[Table]
(cid:12)(cid:12)(cid:12)(cid:12)
[Table]
[Table] [Table]
] (cid:12)(cid:12)(cid:12)
[Table]
(cid:12)(cid:12)(cid:12)
[Equation]
[Table]
[Equation]
Next, we bound the i -th coordinate of W (
[Equation] [Equation]
By independence under Assumption 2, we have that
Var
[Equation]
Using the Gaussian tail bound and union bound
[Equation]
w.p. at least 1 − δ , for ever 1 ≤ i ≤ d
[Equation]
we have that
[Equation]
Since for X ∼ N (0 , σ 2 ) , we have that P ( | X | ≤ t ) ≤
[Equation]
, then for a ﬁxed i ,
[Equation]
Then by union bound, we have that w.p. at least 1 −
[Equation]
4 − 1 , for every 1 ≤ i ≤ d ,
[Equation]
Now deﬁne C 3 := W (0) + 1 − β (cid:107) A (cid:107)
AW
Tand C 4 :
= 1 − β 2 (cid:107) A (cid:107) 2 ˜ r (0)2 i .
We get that C 3 i , i ∈
[ d ] are i.i.d
Gaussian random variables and that C 2 = 12 ( C 3 + C 4 ) , where w.h.p.
for all i ∈ [ d ] , | C 3 i | ≤ ˜ O
(cid:18)d α (cid:19) , | C 3 i | ≥ Θ (cid:18)d 54 α (cid:19) , | C 4 i | ( i ) ≤ ˜ O (cid:18)
d 3 α 2 −(cid:19) , (15) where ( i ) follows from eq.
(14) and the fact that (cid:107) A (cid:107) 2 = √ d .
Then we get that w.h.p.
[Equation]
Substituting eq.
(15) into eq.
(13), we get that w.h.p.,
[Equation]
Similarly, note that
[Equation]
we can use the same techniques to get that i) w.p. at least 1 − δ ,
∀ i ∈ [ d ] : | C 1 i | ≤ ˜ O (cid:0)
[Equation]
ii) w.p.
at least 1 − δ −
[Equation]
C. P ROOF OF L EMMA
The proof in Section C.2 tells us that at the end of the ﬁrst phase (when t = T 1 ),
[Equation]
where v ( T 1 ) T = ηA 1 − β ,
[Equation]
Denote the i -th coordinate of u ( t ) , v ( t ) , R ( t ) 2 as u ( t ) i , v ( t ) i , R ( t ) 2 i , respectively.
Denote the ( i, j ) -th element of R ( t ) 1 as R ( t ) 1 [ i, j ] .
For t ≥ T 1 , we prove by induction that,
[Equation]
17) where
[Equation]
with r ( t ) 1 := η (cid:80) tτ
=
β t − τ (cid:16) W ( t
[Equation] [Equation]
t )
[Equation]
−
A , η t = η (cid:80) tτ =
β t − τ and r ( t ) 2 = η (cid:80)
[Equation]
Note that the r ( t ) 1 and r ( t ) 2 here are different from those deﬁned in Section C.2, but we abuse the notation and still use r ( t ) 1 and r ( t ) 2 to represent the error terms.
The base case is already given by eq.
Suppose our lemma holds for t , then for t + 1 , using the same techniques as in eq.
(5) and eq.
(6), we have that
[Equation]
Plugging in the inductive hypothesis yields
[Equation]
It implies that our lemma holds for t + 1 , which completes the proof.
Now we analyze the error terms (cid:12)(cid:12)(cid:12) R ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12) R ( t )
i (cid:12)(cid:12)(cid:12) .
Eq.
(16) tells us that c ( T 1 ) and ∀ i ∈
[ d ] , v ( T 1 ) i are all positive.
We ﬁrst prove by induction that for all T 1 ≤ t ≤ T 2 , c ( t ) > 0 , ∀ i ∈
[ d ] , v ( t ) i > 0 .
The above discussion already proves the base case.
Suppose at time t , we have c ( t ) > 0 , ∀ i ∈
[ d ] , v ( t ) i > 0 .
Note that when T 1 ≤ t < T 2 , ∀ i ∈ [ d ] : E ( t ) i ≤ 0 , then for t + 1 , v ( t +1)
i = v ( t ) i − η t c ( t ) E ( t ) i > 0 , c ( t +1) =
c ( t ) − η t d (cid:88) i =
E ( t ) i v ( t ) i > 0 .
Therefore by induction, we have proved that for all T 1 ≤ t ≤ T 2 , c ( t ) > 0 , ∀
i ∈
[ d ] , v ( t ) i > 0 .
Now we prove that for all T 1 ≤ t ≤ T 2 , ∀ 1 ≤ i, j ≤ d :
[Equation]
[Equation]
, j ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12)
[Equation]
( T 1 ) i (cid:12)(cid:12)(cid:12) v ( t )
j ≤ δ i + t − 1 (cid:88) τ = T(cid:15)
( τ ) i , 0 ≤ (cid:12)(cid:12)(cid:12)
R ( t ) 2 i (cid:12)(cid:12)(cid:12) c ( t ) (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12) ≤ δ i + t − 1 (cid:88) τ = T(cid:15)
( τ ) i , (18)
where δ i :=
max   max j (cid:12)(cid:12)(cid:12) R ( T 1 )[ i, j ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12) v ( T 1 ) j , (cid:12)(cid:12)(cid:12) R ( T 1 ) 2 i (cid:12)(cid:12)(cid:12) c ( T 1 ) (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12)   , (cid:15) ( t ) i := max   max j (cid:12)(cid:12)(cid:12) r ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) u
( T 1 ) i (cid:12)(cid:12)(cid:12) v ( t ) j , (cid:12)(cid:12)(cid:12) r ( t ) 2 i (cid:12)(cid:12)(cid:12) c ( t ) (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12)   .
are trivial since we have proved that c ( t ) > 0 , ∀ i ∈
[ d ] , v ( t ) i >for all T 1 ≤ t ≤ T 2 .
Now we prove the right hand sides by induction.
The base case is already veriﬁed by the deﬁnition of δ i .
Suppose eq.(18) holds for T 1 ≤ t < T 2 .
Then for t + 1 , using ∀ i ∈
[ d ] : E ( t ) i ≤ 0 and v ( t +1) ≥ v ( t ) , c ( t +1) ≥ c ( t ) , we can get that ∀ 1 ≤ i, j ≤ d (cid:12)(cid:12)(cid:12) R ( t +1)
[ i, j ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) u
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
≤ (cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation] (cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12) v ( t ) j ≤ (cid:16)
δ i + (cid:80) t − 1 τ = T 1 (cid:15) ( τ ) i (cid:17) v ( t ) j + η t (cid:16) δ i + (cid:80) t − 1 τ = T 1 (cid:15) ( τ ) i (cid:17) c
( t ) (cid:16) − E ( t ) j (cid:17) v ( t )
j + η t c ( t ) (cid:16) − E ( t ) j (cid:17) + (cid:15) ( t ) i =
δ i + t (cid:88) τ = T(cid:15)
( τ ) i .
Similarly, we have that ∀ 1 ≤ i ≤ d (cid:12)(cid:12)(cid:12) R ( t +1)
i (cid:12)(cid:12)(cid:12) c ( t +1) (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12) R ( t )
i (cid:12)(cid:12)(cid:12) 1 (cid:12)(cid:12)(cid:12) u ( T 1) i (cid:12)(cid:12)(cid:12) + η t (cid:80) dj =1 (cid:16) − E ( t ) j (cid:17) (cid:12)(cid:12)(cid:12) R ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) 1 (cid:12)(cid:12)(cid:12) u ( T 1) i (cid:12)(cid:12)(cid:12) c
( t ) + η t (cid:80) dj
=1 (cid:16) − E ( t ) j (cid:17) v ( t ) j + (cid:12)(cid:12)(cid:12)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Therefore by induction, eq.
(18) holds for all t in the second phase.
So far we have proved the rank 1 structure stated in Lemma 2.
The remaining part of the proof is given by the following lemma, whose proof is deferred to Section C.9.
Lemma 8.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 d α/ 2+1 .
By picking η ≤ O (cid:16) (cid:15) d 7 α 4 + (cid:17) , we have that w.h.p.
for T 1 ≤ t ≤ min { T 2 , T 3 } , ∀ 1 ≤ i, j ≤ d : 0 ≤ (cid:12)(cid:12)(cid:12) R ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) u
( T 1 ) i (cid:12)(cid:12)(cid:12) v ( t ) j ≤ ˜ O
( (cid:15) 0 ) , 0 ≤ (cid:12)(cid:12)(cid:12) R ( t )
i (cid:12)(cid:12)(cid:12) c ( t ) (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12) ≤ ˜ O ( (cid:15) 0 ) , (19) and that when t = min { T 2 , T 3 } , we have (cid:13)(cid:13) E ( t ) (cid:13)(cid:13) 2 2 = O ( (cid:15) 0 d ) .
C. P ROOF OF L EMMA
We ﬁrst have the following lemma which describes the structure of v ( t ) for t ≥ T 1 .
Lemma 9.
Under Assumption 1, 2 and 3, for t ≥ T 1 , we can write v ( t ) T as v ( t ) T = a ( t ) A + R ( t ) T v , with a ( T 1 ) =
η 1 − β , R ( T 1 ) T v =
[0 , 0 , ..., 0] , and a ( t +1) =
(cid:16) 1 − η t c ( t ) d
( t ) (cid:17) a ( t ) + η
[Equation]
where d ( t ) :
= c ( t ) (cid:13)(cid:13)(cid:13) u ( T 1 ) (cid:13)(cid:13)(cid:13) 2 + R ( t )
Tu ( T 1 ) ,
R ( t ) T:= c ( t
[Equation]
( T 1 ) T R ( t ) 1 + R ( t ) T
[Equation]
( t ) 1 .
Moreover, we have that
[Equation]
We prove Lemma 8 by induction.
Denote the i -th coordinate of R ( t ) 3 and R ( t ) v as R ( t ) 3 i and R ( t ) vi , respectively.
The following lemmas constitute the inductive part.
Lemma 10.
Under Assumption 1, 2 and 3, suppose σ ≤ η
[Equation]
d α/ 2+1 and pick η ≤ O (cid:0) 1 d α (cid:1) .
Consider any t such that T 1 ≤ t < min { T 2 , T 3 } .
Suppose for all T 1 ≤ τ ≤ t , we have ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) w ( τ ) 2 i (cid:12)(cid:12)(cid:12) ≤ O (cid:0) d 1 / 4 (cid:1) , (cid:12)(cid:12)(cid:12) W ( τ ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) ≤ O (cid:0) 1 d 1 / 4 (cid:1) , then we have that ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) r ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:0) η 2 d 11 / 4 (cid:1) , (cid:12)(cid:12)(cid:12) r ( t ) 2 i (cid:12)(cid:12)(cid:12) =
˜ O (cid:0) η 2 d 13 / 4 (cid:1) .
Moreover, we can get that ∀ i ∈
[ d ] : (cid:15) ( t ) i =
˜ O (cid:16)
η 2 d 34 α + 134 (cid:17) , where (cid:15) ( t ) i is deﬁned in eq.
Lemma 11.
Under the conditions of Lemma 10 and pick η ≤ O (cid:16) (cid:15) d 7 α 4 + (cid:17) , we have that at time t + 1 ,
[Equation]
[Equation] [Equation]
( t +
[Equation]
(cid:12) (cid:12)(cid:12) u
( T
[Equation]
i (cid:12) (cid:12)(cid:12) ≤ ˜ O
( (cid:15) 0 ) .
where (cid:15) 0 is deﬁned in Deﬁnition 2.
Lemma 12.
Under the conditions of Lemma 10 and pick η ≤ O (cid:16) (cid:15) d 7 α 4 + (cid:17) , we have that at time t + 1 ,
[Equation]
Moreover,
[Equation]
[Equation] [Equation]
Lemma 13.
Under the conditions of Lemma 10 and pick η ≤ O (cid:16)
[Equation]
(cid:17) , if we further suppose that ∀ j ∈ [ d ] : v ( t ) j c ( t ) =
Θ (cid:16) 1 √ d (cid:17) , (cid:12)(cid:12)(cid:12) R ( t ) 3 j (cid:12)(cid:12)(cid:12)
A j and (cid:12)(cid:12)(cid:12) R ( t ) vj
(cid:12)(cid:12)(cid:12) a ( t )
A j are of order ˜ O
( (cid:15) 0 ) , then we have that at time t + 1 ,
(A) ∀ i, j ∈
[ d ] : E ( t ) j E ( t )
i
[Equation]
, (B) ∀ j ∈ [ d ] : v ( t +1)
j c ( t +1) =
Θ (cid:16) 1 √ d (cid:17) , (C) ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) w ( t +1) 2 i (cid:12)(cid:12)(cid:12)
[Equation] [Equation]
(cid:1)
[Equation]
(D) ∀
j ∈ [ d ] , (cid:12)(cid:12)(cid:12) R ( t +1)
j (cid:12)(cid:12)(cid:12) A j and (cid:12)(cid:12)(cid:12) R ( t +1) vj (cid:12)(cid:12)(cid:12) a ( t +1)
A j are of order ˜ O
( (cid:15) 0 ) .
By combining Lemma 10, 11 and 13, we can prove by induction that for all T 1 ≤ t ≤ min { T 2 , T 3 } , eq.
(19) holds (which follows from Lemma 11), and
[Equation]
i,
[Equation]
∈ [ d ] : E (
[Equation]
Θ(1) , (22) which follows from the part (A) of Lemma 13.
Now the only thing to verify is the base case, i.e. when t = T 1 .
More speciﬁcally, we want to prove that 1) ∀ i, j ∈ [
[Equation]
O (cid:0) d 1 / 4 (cid:1) , (cid:12)(cid:12)(cid:12) W ( T 1 )[ i, j ] (cid:12)(cid:12)(cid:12) ≤
O (cid:0) 1 d 1 / 4 (cid:1) and that 2) ∀
j ∈ [ d ] : v ( T 1)
j c ( T 1) =
Θ (cid:16) 1 √ d (cid:17) , and that 3) (cid:12)(cid:12)(cid:12) R ( T 1) 3 j (cid:12)(cid:12)(cid:12) A j and (cid:12)(cid:12)(cid:12) R ( T 1) vj (cid:12)(cid:12)(cid:12) a ( T 1)
A j are of order ˜ O
( (cid:15) 0 ) .
All of them can be veriﬁed by the proof in Section C.2 and the deﬁnition of R ( t ) v , R ( t ) 3 .
So far we have proved eq.
(19) in Lemma 8.
Now let’s prove when t = min { T 2 , T 3 } , we have that (cid:13)(cid:13) E ( t ) (cid:13)(cid:13)
=
O ( (cid:15) 0 d ) .
If min { T 2 , T 3 } = T 3 , by Deﬁnition 3, we have (cid:13)(cid:13) E ( t ) (cid:13)(cid:13) 2 2 ≤ (cid:15) .
If min { T 2 , T 3 } = T 2 , by Deﬁnition 2, there exists j ∈ [ d ] such that E ( t ) j = − Θ (cid:0) √ (cid:15) 0 (cid:1) .
Combining with eq.
(22) gives us ∀
i ∈
[ d ] :
E ( t ) i = − Θ (cid:0) √ (cid:15) 0 (cid:1) .
Combining these two cases, we get that when t = min { T 2 , T 3 } , (cid:13)(cid:13) E ( t ) (cid:13)(cid:13) 2 2 ≤ max { (cid:15), Θ ( (cid:15)
d ) } =
O ( (cid:15) 0 d ) .
C. P ROOF OF L EMMA
We prove this lemma by induction.
The base case ( t = T 1 ) of v ( t ) is veriﬁed by eq.
Suppose at time t , v ( t ) T = a ( t ) A + R ( t ) T v , then by eq.
17, we have that
W ( t ) 2 W ( t ) 1 = (cid:16) c ( t ) u
[Equation]
T
[Equation]
T
[Equation]
R
[Equation]
t
[Equation]
T
[Equation]
=
[Equation]
u
[Equation]
R ( t )=
d ( t ) v ( t ) T + R ( t ) T= d ( t ) a ( t ) A + d ( t ) R ( t ) T v + R ( t ) T, where d ( t ) :
=
c ( t ) (cid:13)(cid:13) u ( T 1 ) (cid:13)(cid:13) 2 + R ( t )
Tu ( T 1 ) ,
R ( t ) T:=
c ( t ) u ( T 1 ) T R ( t ) 1 + R ( t ) T
R ( t ) 1 .
That gives us v ( t +1) T = v ( t ) T − η t c ( t ) E ( t ) = a ( t ) A + R ( t ) T v − η t c ( t ) (cid:16) d ( t ) a ( t
[Equation]
Therefore we have proved by induction
that for t in the second phase, v ( t ) = a ( t ) A + R ( t ) T v .
The above steps also proved eq.
Write r ( t ) 1 = q ( t ) 11 + q ( t ) 12 where we have q ( t ) 11 = η (cid:80) tτ
=
β t − τ (cid:16) W ( t ) T
E ( t ) − W ( τ ) T
E ( τ ) (cid:17) , q ( t ) 12 = − η (cid:80) tτ
=
β t − τ Dg ( τ ) 1 .
Write r ( t ) 2 = q ( t ) 21 + q ( t ) 22 where q ( t ) 22 = − η (cid:80) tτ
=
β t − τ Dg ( τ ) 2 , q ( t ) 21 = η (cid:80) tτ
=
β t − τ (cid:16) E ( t ) W ( t ) T− E ( τ ) W ( τ ) T(cid:17) .
Let’s ﬁrst bound (cid:12)(cid:12)(cid:12) q ( t ) 11 [ i, j ] (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12) q ( t ) 21 ,i (cid:12)(cid:12)(cid:12) .
By deﬁnition of T 2 , we know that for T 1 ≤ τ ≤ t , ∀ i ∈
[ d ] : (cid:12)(cid:12)(cid:12) E ( τ ) i (cid:12)(cid:12)(cid:12) =
O (1) .
Then we have for all i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) W ( τ +1)
[ i, j ] − W ( τ ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) ≤ η τ (cid:88) k
= β τ − k (cid:12)(cid:12)(cid:12) w ( k )
i E ( k ) j (cid:12)(cid:12)(cid:12)
[Equation]
Note that
[Equation]
We can further get that for ∀ j ∈ [ d ] ,
[Equation]
≤ ηd O (cid:16) d 3 / 4 (cid:17) O (cid:16)
d − 1 / 4 (cid:17)
+ ηd O (cid:16)
d 1 / 4 (cid:17) O (cid:16) d 1 / 4 (cid:17) + η 2 d O (cid:16) d 3 / 4 (cid:17) O (cid:16) d 1 / 4 (cid:17) =
O (cid:16) ηd 3 / 2 + η 2 d 2 (cid:17) =
O (cid:16) ηd 3 / 2 (cid:17) .
Combining the above inequalities gives us ∀ i, j ∈ [ d ] ,
[Equation]
≤
[Equation]
τ = β t − τ ( t − τ ) (cid:16) O (cid:16) d 3 / 4 (cid:17) O (1) + O (cid:16) d 1 / 4 (cid:17) O (cid:16) d 3 / 2 (cid:17)(cid:17) =
O (cid:16) η 2 d 7 / 4 (cid:17) , (cid:12)(cid:12)(cid:12) q ( t )
[Table]
[Table]
(cid:12)(cid:12)(cid:12) =
η (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Table]
(cid:88)
[Table]
β t
[Table]
d (cid:88)
[Table]
(cid:16) E
[Table]
t ) j W ( t )
[ i, j ] − E
( τ ) j W (
[Table]
) 1 [ i, j ] (cid:17)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Table]
(cid:16)(cid:12)(cid:12)(cid:12) E
[Table]
j (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] − W
( τ )
[ i, j ] (cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12) E ( t )
j − E ( τ ) j (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) W ( τ )
[ i, j ] (cid:12)(cid:12)(cid:12)(cid:17)
[Table]
β
[Table]
1 / 4 (cid:17) + O (cid:16) d 3 / 2 (cid:17) O (cid:16)
d − 1 / 4 (cid:17)(cid:17) =
O (cid:16) η 2 d 9 / 4 (cid:17) .
Next let’s bound (cid:12)(cid:12)(cid:12) q ( t ) 12 [ i, j ] (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12) q ( t ) 22 ,i (cid:12)(cid:12)(cid:12) .
By the assumption of this lemma and the analysis before
T 1 , we know that for all τ ≤ t , the M ( τ ) 1 , M ( τ )in Lemma 31 are upper bounded by O (cid:0) 1 d 1 / 4 (cid:1) and
O (cid:0) d 1 / 4 (cid:1) , respectively.
In the theorem we consider the training period before T SGD , 2 so the time T in
Lemma 31 is set as T SGD , 2 .
In the following sections, we will prove that T SGD
[Equation]
Then by Lemma 31, we have with probability at least 1 − 1 d , for ∀ τ ≤ t and ∀ i, j ∈ [ d ] ,
[Equation]
By picking σ ≤ η 3 / 2 d α/ 2+1 , we have (cid:12)(cid:12)(cid:12) Dg ( τ )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ η ˜ O (cid:16) d 114 (cid:17) and (cid:12)(cid:12)(cid:12) Dg ( τ )
i (cid:12)(cid:12)(cid:12) ≤ η ˜ O (cid:16) d 134 (cid:17) , which yields (cid:12)(cid:12)(cid:12) q ( t ) 12 [ i, j ] (cid:12)(cid:12)(cid:12) ≤ η t (cid:88) τ
= β t − τ (cid:12)(cid:12)(cid:12) Dg ( τ )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16)
η 2 d 114 (cid:17) , (cid:12)(cid:12)(cid:12) q ( t ) 22 ,i (cid:12)(cid:12)(cid:12) ≤ η t (cid:88) τ
= β t − τ (cid:12)(cid:12)(cid:12) Dg ( τ )
i (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) η 2 d 134 (cid:17) .
Combining the above bounds, we get that ∀
i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) r ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16)
η 2 d 114 (cid:17) , (cid:12)(cid:12)(cid:12) r ( t ) 2 i (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) η 2 d 134 (cid:17) .
By the analysis in Section C.2, we know that at time T 1 , for some i 0 ∈
[ d ] , c ( T 1 ) (cid:12)(cid:12)(cid:12) u ( T 1 ) i 0 (cid:12)(cid:12)(cid:12) =
Θ (cid:16)
d α 2 (cid:17) , and for ∀ i, j ∈ [ d ] , we have c ( T 1 ) (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12) =
˜Ω (cid:16)d 3 α(cid:17) and (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12) v ( T 1 )
j = ˜Ω (cid:16) d 3 α 4 + (cid:17) , which gives us ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) r ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12) (cid:12) u ( T 1 ) i (cid:12)(cid:12) (cid:12) v ( t ) j ≤ (cid:12)(cid:12)(cid:12) r ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12) (cid:12) u ( T 1 ) i (cid:12)(cid:12) (cid:12) v ( T 1 )
j = ˜ O (cid:16)
η 2 d 34 α + 134 (cid:17) , (cid:12)(cid:12)(cid:12) r ( t ) 2 i (cid:12)(cid:12)(cid:12) c ( t ) (cid:12)(cid:12) (cid:12) u ( T 1 ) i (cid:12)(cid:12) (cid:12) ≤ (cid:12)(cid:12)(cid:12) r ( t ) 2 i (cid:12)(cid:12)(cid:12) c ( T 1 ) (cid:12)(cid:12) (cid:12) u ( T 1 ) i (cid:12)(cid:12) (cid:12) =
˜ O (cid:16) η 2 d 34 α + 134 (cid:17) .
Hence we get the bound ∀ i ∈
[ d ] : (cid:15) ( t ) i ≤ ˜ O (cid:16) η 2 d 34 α + 134 (cid:17) .
C. P ROOF OF L EMMA
Let’s ﬁrst try to bound the length of min { T 2 , T 3 } .
More formally, we prove that under the conditions of Lemma 10 and pick η ≤ O (cid:16) (cid:15)
d 7 α 4 + (cid:17) , we have that min { T 2 , T 3 } ≤ O (cid:18)
d α log( √ d/(cid:15) ) η (cid:19) .
Under the conditions of Lemma 10, we know that ∀ j ∈ [ d ] : (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) r ( t ) 2 W ( t ) 1 (cid:17) j (cid:12) (cid:12)(cid:12)(cid:12) ≤ d (cid:88) i =
(cid:12)(cid:12)(cid:12) r ( t )
i W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
O (cid:0) η 2 d 4 (cid:1) , (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) W ( t ) 2 r ( t ) 1 (cid:17) j (cid:12)(cid:12)(cid:12)(cid:12) ≤
d (cid:88) i = (cid:12)(cid:12)(cid:12) w ( t )
i r ( t )
[Table] [Table]
[Table] [Table]
Combining with
, we get
[Equation]
Then we have
[Equation]
When T 1 ≤ t < T 2 , we have proved that c ( t ) is increasing over time in Section C.8, which implies that (cid:13)(cid:13)(cid:13) W
[Equation]
[Equation]
C (cid:13)(cid:13)(cid:13) W
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)2 since c ( t ) u ( T 1 ) T is the leading term of W ( t ) 2 .
Combining with η t ≥ η gives us
[Equation]
⇒
[Table]
[Equation]
[Table]
(cid:13)(cid:13)(cid:13) + (cid:18) 1 − ηC (cid:13)(cid:13)(cid:13) W ( T 1 )(cid:13)(cid:13)(cid:13) (cid:19) t − T 1  (cid:13)(cid:13)(cid:13) E
( T 1 ) (cid:13)(cid:13)(cid:13) 2 −
O (cid:0) η 2 d 4 (cid:1) ηC (cid:13)(cid:13)(cid:13) W ( T 1 )(cid:13)(cid:13)(cid:13) 
 ( i ) ≤ O   ηd(cid:13)(cid:13)(cid:13)
W ( T 1 )(cid:13)(cid:13)(cid:13) 
 + exp (cid:18) − ηC (cid:13)(cid:13)(cid:13) W ( T 1 )(cid:13)(cid:13)(cid:13)2 ( t − T 1 ) (cid:19)
O (cid:16) √ d (cid:17) , where ( i ) uses (cid:13)(cid:13) E ( T 1 ) (cid:13)(cid:13) 2 =
O ( √ d ) .
By picking η ≤ O (cid:16) (cid:15) d 7 α 4 + (cid:17) and noticing that (cid:13)(cid:13)(cid:13) W ( T 1 )(cid:13)(cid:13)(cid:13)2 ≥ Ω (cid:0)
d α (cid:1) , we have ηd(cid:13)(cid:13)(cid:13)
W ( T 1)(cid:13)(cid:13)(cid:13) 2 2 < √ (cid:15) 2 .
Hence when t − T 1 ≥ Θ (cid:32) log (cid:16) √ d/(cid:15) (cid:17) η (cid:13)(cid:13)(cid:13) W ( T 1)(cid:13)(cid:13)(cid:13) 2(cid:33) , we have that (cid:13)(cid:13) E ( t ) (cid:13)(cid:13) 2 ≤ √ (cid:15) , i.e. (cid:13)(cid:13) E ( t ) (cid:13)(cid:13) 2 2 ≤ (cid:15) .
That means after at most O (cid:32) log (cid:16) √ d/(cid:15) (cid:17) η (cid:13)(cid:13)(cid:13) W ( T 1)(cid:13)(cid:13)(cid:13)
2(cid:33) steps from T 1 , either t ≥ T 2 , or we have (cid:13)(cid:13) E ( t ) (cid:13)(cid:13) 2 2 ≤ (cid:15) .
In other words, min { T 2 , T 3 } ≤ T 1 + O (cid:32) log (cid:16) √ d/(cid:15) (cid:17) η (cid:13)(cid:13)(cid:13) W ( T 1)(cid:13)(cid:13)(cid:13) 2(cid:33)
≤ O (cid:18)
d α log (cid:16) √ d/(cid:15) (cid:17) η (cid:19) .
Now we are ready to bound eq.
Combining min { T 2 , T 3 } ≤ O (cid:18)
d α log( √ d/(cid:15) ) η (cid:19) and Lemma 10 yields that for t + 1 ≤ min { T 2 , T 3 } , ∀ i ∈
[ d ] , t +1 (cid:88) τ = T(cid:15)
[Equation]
Lemma 1 tells us that δ i =
˜ O (cid:16) d 1 4 α −(cid:17) .
Substituting these bounds into eq.
(18) completes the proof.
C. P ROOF OF L EMMA
The proof in Section C.8 tells us that for T 1 ≤ τ ≤ T 2 , c ( τ ) > 0 , ∀ j ∈ [ d ] : v ( τ ) j > 0 , which gives us 0 ≤ (cid:12)(cid:12)(cid:12) R ( t +1)
Tu ( T 1) (cid:12)(cid:12)(cid:12) c
( t +1) (cid:107) u ( T 1) (cid:107) 2 and 0 ≤ (cid:12)(cid:12)(cid:12) R ( t +1) 3 j (cid:12)(cid:12)(cid:12) c ( t +1) (cid:107) u
( T 1) (cid:107) 2 v ( t +1)
j .
By Lemma 11, we have that ∀ 1 ≤ i, j ≤ d : 0 ≤ (cid:12)(cid:12)(cid:12) R
( t +1)
[ i, j ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) u
( T 1 ) i (cid:12)(cid:12)(cid:12) v ( t +1)
j ≤ ˜ O ( (cid:15) 0 ) , 0 ≤ (cid:12)(cid:12)(cid:12) R ( t +1) 2 i (cid:12)(cid:12)(cid:12) c ( t +1) (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12) ≤ ˜ O ( (cid:15) 0 ) , which gives us (cid:12)(cid:12)(cid:12) R ( t +1)
Tu ( T 1 ) (cid:12)(cid:12)(cid:12) c
( t +1) (cid:13)(cid:13)
u ( T 1 ) (cid:13)(cid:13)
[Table]
Lemma 9 tells us that
R ( t +1)
T= c ( t +1) u ( T 1 ) T R (
[Table]
+1)+ R ( t +1)
[Table]
R ( t +1).
And we have that (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) c ( t
[Table] [Table]
u
[Table] [Table]
R ( t
[Table]
(cid:17) j (cid:12)(cid:12)(cid:12)(cid:12) c
[Table]
v
[Table]
≤ c
[Table]
O (cid:0) (cid:15) 20 (cid:1) c
( t +1) (cid:80) di =1 (cid:12)(cid:12)(cid:12) u
( T 1 ) i (cid:12)(cid:12)(cid:12) 2 v ( t +1)
j c ( t +1) (cid:80) di
=1 (cid:12)(cid:12)(cid:12) u
( T 1 ) i (cid:12)(cid:12)(cid:12) 2 v ( t +1)
j = ˜ O (cid:0) (cid:15) 20 (cid:1) .
Therefore (cid:12)(cid:12)(cid:12) R ( t +1) 3 j (cid:12)(cid:12)(cid:12) c ( t +1) (cid:13) (cid:13) u ( T 1 ) (cid:13) (cid:13) 2 v ( t +1)
j ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) c ( t +1) u ( T 1 ) T R ( t +1)(cid:17) j (cid:12)(cid:12)(cid:12)(cid:12) c
( t +1) (cid:13) (cid:13) u
( T 1 ) (cid:13) (cid:13) 2 v ( t +1)
j + (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) R ( t +1) T
R ( t +1)(cid:17) j (cid:12)(cid:12)(cid:12)(cid:12) c
( t +1) (cid:13) (cid:13) u
( T 1 ) (cid:13) (cid:13) 2 v ( t +1)
j ≤ ˜ O ( (cid:15) 0 ) .
By Lemma 9,
[Equation]
u ( T 1 ) (cid:13)(cid:13)(cid:13) 2 v ( t +1)
T + R ( t +1)
Tu ( T 1 ) v ( t +1) T + R
( t +1)
T.
Then we have that ∀ j ∈
[ d ] , (cid:16) W ( t +1)
W
[Equation]
t +1)
[Equation]
(cid:17)
[Equation]
( t +
[Equation]
(cid:16)
+ e ( t +1) j (cid:17) , where (cid:12)(cid:12)(cid:12) e ( t +1) j (cid:12)(cid:12)(cid:12) ≤ ˜ O
( (cid:15) 0 ) .
Since t < T 2 , we have ∀ j ∈
[ d ] : (cid:16)
[Equation]
( t +1)
[Equation]
( t +1)(cid:17)
j A j = O (1) , which yields
[Equation]
[Equation]
(25) which proves eq.
(21), since ∀ j ∈ [ d ] :
[Equation]
C. P ROOF OF L EMMA
(A) Under the conditions of Lemma 10 and pick η ≤ O (cid:16) (cid:15)
d 7 α 4 + (cid:17) , we can apply the technique when proving eq.
(24) to show that eq.
(24) also holds at time t .
Since (cid:12)(cid:12)(cid:12) R ( t ) vj (cid:12)(cid:12)(cid:12) a ( t ) A j ≤ ˜ O
( (cid:15) 0 ) , we get that v ( t
[Equation]
(cid:16)
+ e ( t ) vj (cid:17) , where (cid:12)(cid:12)(cid:12) e ( t ) vj (cid:12)(cid:12)(cid:12) ≤ ˜ O
( (cid:15) 0 ) .
Substituting into the time t version of eq.(24) yields ∀ j ∈
[ d ] : (cid:16) W ( t ) 2 W ( t ) 1 (cid:17) j = a ( t ) c
( t ) (cid:13)(cid:13)(cid:13) u ( T 1 ) (cid:13)(cid:13)(cid:13)
A j (cid:16)
+ ˜ e ( t ) j (cid:17) , where (cid:12)(cid:12)(cid:12) ˜
e ( t ) j (cid:12)(cid:12)(cid:12) ≤ ˜ O
( (cid:15) 0 ) ,
That gives us
[Equation]
2 ˜ e ( t ) j (cid:19) .
Since t < T 2 , we have E ( t ) j < −√ (cid:15) 0 .
Combining with A j = Θ(1) , gives us a ( t ) c ( t ) (cid:13) (cid:13) u ( T 1 ) (cid:13) (cid:13) 2 − 1 = − Ω (cid:0) √ (cid:15) 0 (cid:1) .
Then we can rewrite E ( t ) j as ∀ j ∈
[ d ] ,
E ( t ) j
= A j (cid:18) a ( t ) c
( t ) (cid:13)(cid:13)(cid:13) u ( T 1 ) (cid:13)(cid:13)(cid:13) 2 − 1 (cid:19) (cid:32)
+ a ( t ) c
( t ) (cid:13)(cid:13) u
( T 1 ) (cid:13)(cid:13)a ( t ) c ( t ) (cid:13)(cid:13) u
( T 1 ) (cid:13)(cid:13) 2 − 1 ˜
e
( t ) j (cid:33) :
= A j (cid:18) a ( t ) c ( t ) (cid:13)(cid:13)(cid:13) u ( T 1 ) (cid:13)(cid:13)(cid:13) 2 − 1 (cid:19) (cid:16) 1 + e ( t ) Ej (cid:17) , where (cid:12)(cid:12)(cid:12) e ( t ) Ej (cid:12)(cid:12)(cid:12) =
˜ O ( √ (cid:15) 0 ) .
Hence ∀ i, j ∈ [ d ] : E ( t ) j E ( t ) i = Θ(1) .
(B) Note that we assume ∀ j ∈ [ d ] : v ( t ) j c ( t ) =
Θ (cid:16) 1 √ d (cid:17) , then we have for j ∈ [ d ] , − E ( t ) v ( t ) c ( t ) (cid:16) − E ( t ) j (cid:17) =
(cid:80) di
=1 (cid:16) − E ( t ) i (cid:17) v ( t ) i c ( t ) (cid:16) − E ( t ) j (cid:17) =
d (cid:88) i =
E ( t )
i
E ( t ) j · v ( t )
i c ( t ) =
d (cid:88)
i =
Θ (cid:18) 1 √ d (cid:19) =
Θ (cid:16) √ d (cid:17) , ⇒ c ( t ) (cid:16) − E ( t ) j (cid:17) − E ( t ) v ( t ) =
Θ (cid:18) 1 √ d (cid:19) .
Then for t + 1 , we have that for j ∈ [ d ] , v ( t +1)
j c ( t +1) =
v ( t ) j + η t c ( t ) (cid:16) − E ( t ) j (cid:17)
c
( t ) + η t (cid:0) − E ( t ) (cid:1) v ( t ) =
Θ (cid:18) 1 √ d (cid:19) .
(C)
Combining eq.
(25) and ∀ j ∈ [ d ] : A j = Θ(1) , we know that c ( t +1) (cid:13)(cid:13)(cid:13) u ( T 1 ) (cid:13)(cid:13)(cid:13) 2 v ( t +1)
j ≤ O (1) , which yields ∀ j ∈ [ d ] , (cid:13)(cid:13)(cid:13) u ( T 1 ) (cid:13)(cid:13)(cid:13) 2 (cid:16) v ( t +1) j (cid:17) 2 ≤ v ( t +1)
j c ( t +1) O
(1) =
O (cid:18) 1 √ d (cid:19) , (cid:16) c ( t +1) (cid:17) 2 (cid:13)(cid:13)(cid:13) u
( T 1 ) (cid:13)(cid:13)(cid:13) 2 ≤ c ( t +1) v ( t +1)
j
O (1) =
O (cid:16) √ d (cid:17) .
Hence ∀ i, j ∈ [ d ] , c ( t +1) (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12) =
O (cid:16) d 1 / 4 (cid:17) ⇒ (cid:12)(cid:12)(cid:12) w ( t +1)
i (cid:12)(cid:12)(cid:12) ≤ c ( t +1) (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12) R ( t +1) 2 i (cid:12)(cid:12)(cid:12) ( i ) =
O (cid:16) d 1 / 4 (cid:17) , (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12) v ( t +1)
j = O (cid:18)d 1 /(cid:19)
⇒ (cid:12)(cid:12)(cid:12) W ( t +1)
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12) u ( T 1 ) i (cid:12)(cid:12)(cid:12) v ( t +1)
j + (cid:12)(cid:12)(cid:12) R ( t +1)
[ i, j ] (cid:12)(cid:12)(cid:12)
[Equation]
where ( i ) and ( ii ) use Lemma 11.
(D)
The fact that ∀ j ∈ [ d ] : (cid:12)(cid:12)(cid:12) R ( t +1)
3 j (cid:12)(cid:12)(cid:12)
A j ≤ ˜ O ( (cid:15) 0 ) was already proved in Lemma 12 in eq.(21).
To analyze (cid:12)(cid:12)(cid:12) R ( t +1) vi (cid:12)(cid:12)(cid:12) a ( t +1)
A i , we ﬁrst prove that 1 − η t c ( t ) d ( t ) > 0 .
It is not hard to prove that eq.(26) also holds for time t .
Recall that d ( t ) =
c ( t ) (cid:13)(cid:13) u ( T 1 ) (cid:13)(cid:13) 2 + R ( t )
Tu ( T 1 ) and Lemma 12 tells us that 0 ≤ (cid:12)(cid:12)(cid:12) R ( t )
Tu ( T 1) (cid:12)(cid:12)(cid:12) c
( t ) (cid:107) u ( T 1) (cid:107) 2 ≤
˜ O
( (cid:15) 0 ) , then we have c ( t ) d ( t ) = (cid:16) c ( t ) (cid:17) 2 (cid:13)(cid:13)(cid:13)
u
( T 1 ) (cid:13)(cid:13)(cid:13)
+ c ( t ) R ( t )
Tu ( T 1 ) ≤ O (cid:16) √ d (cid:17) .
Under the conditions of Lemma 10, and pick η ≤ O (cid:16) (cid:15) d 7 α 4 + (cid:17) , we have that 1 − η t c ( t ) d ( t ) ≥ 1 − ηc ( t ) d ( t ) > 0 .
The assumption ∀ j ∈ [ d ] : (cid:12)(cid:12)(cid:12) R ( t
[Equation]
3 j (cid:12)(cid:12)(cid:12) A
j ≤ ˜ O
( (cid:15) 0 ) together with c ( t ) > 0 gives us η
[Equation]
η
[Equation]
≤ ˜ O
[Equation]
Combining with the assumption (cid:12)(cid:12)(cid:12) R ( t ) vi (cid:12)(cid:12)(cid:12) a ( t ) A i ≤ ˜ O ( (cid:15) 0 ) yields ∀ i ∈
[ d ] : (cid:12)(cid:12)(cid:12) R ( t +1) vi (cid:12)(cid:12)(cid:12) a ( t +1)
A i ≤ (cid:0) 1 − η t c ( t ) d ( t ) (cid:1) (cid:12)(cid:12)(cid:12) R
( t ) v (cid:12)(cid:12)(cid:12) + η t c ( t ) (cid:12)(cid:12)(cid:12) R ( t )
i (cid:12)(cid:12)(cid:12) (cid:0) 1 − η t c ( t ) d ( t ) (cid:1) a ( t ) A i + η t c ( t ) A i ≤ ˜ O ( (cid:15) 0 ) .
D A NALYSIS OF A DAM
Note that A = 1 m Y X T , Λ xx :
= 1 m XX T .
Denote g ( t ) k :=
∇ W k L ( W ( t ) ) , k = 1 , 2 .
We have that g
[Equation]
=
[Equation] [Equation]
, g
[Equation]
(cid:17) W ( t ) T.
Let ˜ A ( t ) , ˜Λ ( t ) xx and ˜ g ( t ) k , k = 1 , 2 be the corresponding batch versions at time t .
Let E ( t ) :=
W ( t ) 2 W ( t ) 1 − A , and denote E ( t ) j as the j -th component of E ( t ) .
We also denote ∆ w ( t ) 2 i :=
w ( t +1) 2 i − w ( t ) 2 i , ∆ W ( t ) 1 [ i, j ] := W ( t +1)
[ i, j ] − W ( t )
[ i, j ] .
By eq.
(2), the update equations of Adam are given by
[Equation]
where Dg ( t ) 1 := ˜ g ( t ) 1 − g ( t ) 1 and Dg ( t ) 2 :=
˜ g ( t ) 2 − g ( t ) 2 , and r ( t ) 1 n [ i, j ] :
= (1 − β 1 ) t (cid:88) τ =
[Equation] [Equation]
[Equation]
Denote the i -th coordinate of W 2 W 1 and A as ( W 2 W 1 ) i and A i , respectively.
By Assumptionand the assumption that ∀ i ∈ [ d ] : A i > 0 , A i = Ω(1) , at the beginning, w.h.p., ∀ i ∈
[ d ] : ( W 2 W 1 ) i − A i < 0 .
Based on this, we divide the training procedure into two phases (note that these two phases are different from those of SGD+M).
First phase : when the error ( W 2 W 1 ) i − A
i is negative and its absolute value is big for all i ∈
[ d ] . 2.
Second phase : when ( W 2 W 1 ) i − A
i is close to zero for some coordinate
i ∈
[ d ] .
More formally, we deﬁne the boundary between the two phases below.
Deﬁnition
(End of the ﬁrst phase).
The end of the ﬁrst phase (denoted as T 1 ) is deﬁned as
T 1 = inf (cid:110) t > 0 : ∃ i ∈ [ d ] : E ( t ) i ≥ −√ ηd (cid:111) .
In the second phase, we deﬁne some time points.
Deﬁnition 5.
Deﬁne T g := inf
[Equation]
For t < T 1 , we have ∀ i ∈
[ d ] : E ( t ) i < 0 by Deﬁnition 4.
For t > T 1 , some E ( t ) i may ﬂip the sign and become positive.
For certain coordinate i , we deﬁne the following “ﬂip time”.
Deﬁnition 6.
Deﬁne T f,i := inf (cid:110) t > T 1 : E ( t ) i ≥ −√ ηd (cid:111) .
Deﬁne T f := max i T f,i as the largest “ﬂip time” over all
i ∈
[ d ] , i.e. the “ﬂip time” of the last E i which ﬂips the sign.
Moreover, denote ˜ T :
=
min { T g , T f } .
We can ﬁrst show that after a few steps in the ﬁrst phase, W 1 will become an approximately rankmatrix, as described in the following lemma.
Lemma 14.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
By picking η ≤ O (cid:0) 1 d 3 α (cid:1) , ξ ≤ (cid:112) η d 3 α − 1 , and β 2 = β 21 , there exists t inc > 0 such that w.h.p.
for t inc ≤ t < T 1 , ∀ i, j ∈ [ d ] : w ( t )
i = sign (cid:16) w (0)2 i (cid:17) η
( t − t inc ) + R ( t )
i ,
W ( t ) 1 [ i, j ] = sign (cid:16) w (0)2 i (cid:17) η ( t − t inc ) + R ( t ) 1 [ i, j ] , where (cid:12)(cid:12)(cid:12) R ( t )
[ i,j ] (cid:12)(cid:12)(cid:12) η ( t − t inc ) =
˜ O (cid:16) √ η + 1 η ( t − t inc ) d α (cid:17) , (cid:12)(cid:12)(cid:12) R ( t ) 2 i (cid:12)(cid:12)(cid:12) η ( t − t inc ) =
˜ O (cid:16) √ η + 1 η ( t − t inc ) d α (cid:17) .
Specially, when t = T 1 , we have that ∀ i, j ∈ [ d ] : w ( T 1 ) 2 i = sign (cid:16) w (0)2 i (cid:17)
η ( T 1 − t inc ) + R ( T 1 ) 2 i ,
W ( T 1 )[ i, j ] = sign (cid:16) w (0)2 i (cid:17)
η ( T 1 − t inc ) + R ( T 1 )[ i, j ] , where η ( T 1 − t inc ) =
Θ (cid:16) 1 √ d (cid:17) and
[Equation]
The following lemma tells us that this approximate rank-1 structure is preserved when T 1 ≤ t ≤ ˜ T .
Lemma 15.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
By picking η ≤ O (cid:0) 1 d 3 α (cid:1) , ξ ≤ (cid:112) η d 3 α − 1 , and β 2 = β 21 , we have w.h.p.
for T 1 ≤ t < ˜ T , ∀ i, j ∈ [ d ] :
[Equation]
t )
i
[Equation]
sign
[Equation] [Equation]
i
[Equation]
( t ) 2 i ,
[Equation]
) 1 [ i, j ] = sign (cid:16) w
[Equation]
0)2 i (cid:17) V ( t )
j
[Equation]
) 1 [ i, j ] , where
[Equation]
i, j ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) V ( t ) j (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:18)
η 14 + d α 2 −(cid:19) , and that L (cid:16) W ( ˜ T ) (cid:17) ≤ ˜ O (cid:0) ηd 4 (cid:1) .
Now we are ready to prove the Adam part of Theorem 1.
D. P ROOF OF THE A DAM PART OF T HEOREM
Deﬁne T Adam , 1 = t inc + 1 ηd α 2 .
Note that this choice of T Adam , 1 gives η ( T Adam , 1 − t inc ) =
d α 2 .
By picking η ≤ O (cid:0) 1 d 3 α (cid:1) , ξ ≤ (cid:112) η d 3 α − 1 and β 2 = β 21 , we can apply Lemma 14 to get that ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) w ( T Adam , 1 ) 2 i (cid:12)(cid:12)(cid:12) =
[Equation]
[ i, j ] (cid:12)(cid:12)(cid:12) =
Θ (cid:16)
d α 2 (cid:17) , and therefore ∀ i ∈ [ d ] : E ( T Adam , 1 ) i = − Θ(1) and L (cid:0) W ( T Adam , 1 ) (cid:1)
=
Θ( d ) .
Deﬁne T Adam , 2 = ˜ T .
By Lemma 15, we have L (cid:0) W ( T Adam , 2 ) (cid:1) =
˜ O (cid:0) ηd 4 (cid:1) .
For any p > 0 , by picking α ≥ p +43 , we have L (cid:0) W ( T Adam , 2 ) (cid:1) =
˜ O (cid:0) ηd 4 (cid:1) ≤ ˜ O
(cid:0) 1 d p (cid:1) .
Moreover, combining Lemma 14 and 15, we get that when t ∈ [ T Adam , 1 , T Adam , 2 ] , the conditions in
Lemma 30 are satisﬁed with δ =
˜ O (cid:16)
η 14 + d α 2 −(cid:17) .
The i -th component of the u vector (denoted as u i ) is sign (cid:16) w (0)2 i (cid:17) .
That means ∀ i ∈
[ d ] : u 2 i = 1 and max i ( u i ) 2 median ( u i ) 2 = 1 .
Then we can apply Lemmaand get that
[Equation]
(cid:34)(cid:18) 1 − δ 1 + δ (cid:19) 2 max i ( u i )median ( u i ) 2 , (cid:18) 1 + δ 1 − δ (cid:19) 2 max i ( u i )median ( u i )(cid:35) = (cid:34) (cid:18) 1 − δ 1 + δ (cid:19), (cid:18) 1 + δ 1 − δ (cid:19) 2 (cid:35) , ⇒
R Adammed , 1 ( t ) , R Adammed , 2 ( t ) = 1 ±
O ( δ ) =
1 ± ˜ O (cid:18)
η 14 + d α 2 −(cid:19) .
D. P ROOF OF L EMMA
For some time t , we introduce two conditions.
Condition 1.
∀ τ ∈
[ H ] : sign (cid:16) g
[Equation]
t − τ )
[ i, j ] (cid:17) =
s ( t )
[ i, j ] , (1 − β 1 ) (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)
H (cid:88) τ = β ( τ ) 1 g ( t − τ )
[ i, j ] (cid:12) (cid:12)(cid:12)(cid:12)(cid:12) ≥ Ω( ξ ) .
Condition 2.
∀ τ ∈
[ H ] : sign (cid:16) g ( t − τ ) 2 i (cid:17) =
s ( t )
i , (1 − β 1 ) (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)
H (cid:88) τ
= β ( τ ) 1 g ( t − τ ) 2 i (cid:12)(cid:12)(cid:12)(cid:12) (cid:12) ≥ Ω( ξ ) .
Next prove that, under Assumption 1 and 2, by picking η ≤ O (cid:0) 1 d 3 α (cid:1) , ξ ≤ (cid:112) η d 3 α − 1 , and β 2 = β 21 , there exists t inc > 0 such that for t inc ≤ t < T 1 , the weights can be approximated in the following way.
[Equation]
where (cid:12)(cid:12)(cid:12) e ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:0) √ η (cid:1) ,
[Equation]
Before we dive into the proof, let’s introduce some useful lemmas.
The following lemma reﬂects our key idea: converting the exponential average in Adam to a ﬁnite-step average, and trying to bound the stochastic error terms in eq.
Lemma 16.
Under Assumption 1, 2 and 3 and pick β= β 21 .
Let M ( t ):=
[Equation]
w ( τ ) 2 i (cid:12)(cid:12)(cid:12) , G ( t ) 1 := max i,j ∈ [ d ] ,τ ≤ t (cid:12)(cid:12)(cid:12) g ( τ ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) and
G ( t ) 2 := max i,j ∈ [ d ] ,τ ≤ t (cid:12)(cid:12)(cid:12) g ( τ ) 2 i (cid:12)(cid:12)(cid:12) .
We have that w.h.p., for all t ≤ ˜ O (cid:16)
[Equation]
dη (cid:17) and ∀ i, j ∈ [ d ] ,
[Equation]
where H ≥
[Equation]
ηξand (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 1 n [ i, j ] (cid:12)(cid:12)(cid:12) ≤
O ( ηξ 2 ) + O (cid:16) D ( t ) 1 (cid:17) , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 1 d [ i, j ] (cid:12)(cid:12)(cid:12) ≤ O ( ηξ 2 ) + O (cid:18) D ( t ) 1 G ( t ) 1 + (cid:16) D ( t ) 1 (cid:17) 2 (cid:19) , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 2 n,i (cid:12)(cid:12)(cid:12) ≤
O ( ηξ 2 ) + O (cid:16) D ( t ) 2 (cid:17) , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 2 d,i (cid:12)(cid:12)(cid:12) ≤ O ( ηξ 2 ) + O (cid:18) D ( t ) 2 G ( t ) 2 + (cid:16) D ( t ) 2 (cid:17) 2 (cid:19) , with
D ( t ) 1 ≤ ˜ O   d 3 M ( t ) 1 (cid:16) M ( t ) 2 (cid:17) 2 σ (cid:115) d 1 /η 

+ ˜
O   M ( t ) 2 σ (cid:115) d 3 /η 
 ,
D ( t ) 2 ≤ ˜ O   d 4 (cid:16) M ( t ) 1 (cid:17) 2 M ( t ) 2 σ (cid:115) d 1 /η 

+ ˜
O   dM ( t ) 1 σ (cid:115) d 3 /η 
 .
Corollary 2.
Under the conditions of Lemma 16 and suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
Consider any t ≤ ˜ O (cid:16) 1 √ dη (cid:17) .
If M ( t ) 1 , M ( t ) 2 ≤ ˜ O (cid:16) 1 √ d (cid:17) , G ( t ) 1 ≤ ˜ O (cid:16) 1 √ d (cid:17) , G ( t ) 2 ≤ ˜ O (cid:16) √ d (cid:17) , then H in Lemma 16 can be picked as 1 1 − β 1 log dηξ 2 and we can get that ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 1 n [ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t )
d [ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 2 n,i (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 2 d,i (cid:12)(cid:12)(cid:12) ≤ ˜ O ( ηξ 2 ) .
The following lemma analyzes the magnitude of weights during a short period at the beginning.
Lemma 17.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
Pick ξ ≤ d 3 2 α , then there exists some time point t inc ∈ ( H, T 1 ) , such that w.h.p., for t ≤ t inc , for every i, j ∈
[ d ] ,
[Equation]
Speciﬁcally, when t = t inc , we have sign (cid:16) w ( t inc ) 2 i (cid:17)
= sign (cid:16) W ( t inc )[ i, j ] (cid:17) =
sign (cid:16) w (0)2 i (cid:17) , (cid:12)(cid:12)(cid:12) W ( t inc )[ i, j ] (cid:12)(cid:12)(cid:12) =
Θ (cid:16) d 32 α + (cid:17) and (cid:12)(cid:12)(cid:12) g ( t inc )[ i, j ] (cid:12)(cid:12)(cid:12) ≥ Ω( ξ ) , (cid:12)(cid:12)(cid:12) g ( t inc ) 2 i (cid:12)(cid:12)(cid:12) ≥ Ω( ξ ) .
Moreover, Conditionand 2 are satisﬁed for t = t inc .
The s ( t ) 1 [ i, j ] and s ( t ) 2 i in the conditions are both − sign (cid:16) w (0)2 i (cid:17) .
The following lemma gives us lower bounds of (cid:12)(cid:12)(cid:12) g ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12) g ( t )
i (cid:12)(cid:12)(cid:12) .
Lemma 18.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
Pick ξ ≤ (cid:112) η
d 3 α − 1 , η ≤ O (cid:0) 1 d 3 α (cid:1) .
Consider t inc in Lemma 17.
We have w.h.p.
for any t ∈ [ t inc , T 1 ) , and for ∀ i, j ∈ [ d ] , sign (cid:16) ∆ W ( t ) 1 [ i, j
[Table]
Moreover
[Table]
The following lemma shows
[Table]
when
[Table]
(cid:12) and that (cid:12)(cid:12)(cid:12) g ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) (cid:29) (cid:12)(cid:12)(cid:12) g ( t )
[ i, j ] − g ( t − 1)
[ i, j ] (cid:12)(cid:12)(cid:12) .
Lemma 19.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
Pick ξ ≤ (cid:112) η
d 3 α − 1 , η ≤ O (cid:0) 1 d 3 α (cid:1) .
For t inc in Lemma 17, we have that w.h.p.
for t inc ≤ t < T 1 and τ ≤ t , ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
(30) (cid:12)(cid:12)(cid:12)(cid:12)(cid:16)
[Equation] [Equation]
] (cid:17)
[Equation]
[Equation] [Equation]
(cid:17)
[Equation]
(cid:16)
[Equation] [Equation]
Equipped with these lemmas, now let’s prove eq.
For any t ∈ [ t inc , T 1 ) , by Lemma 18, we know that M ( t ) 1 , M ( t )≤ ˜ O (cid:16) 1 √ d (cid:17) , and that G ( t )≤ ˜ O (cid:16) 1 √ d (cid:17) , G ( t ) 2 ≤ ˜ O (cid:16) √ d (cid:17) .
At the end of the proof for this lemma, we will show that T 1 = Θ (cid:16) 1 √ dη (cid:17) .
Then we can pick H :
= 1 1 − β 1 log dηξ 2 and apply Lemma 16 and Corollary 2 to get that, w.h.p., for all t ∈ [ t inc , T 1 ) and ∀ i, j ∈
[ d ] , eq. (27) can be written as
[Equation]
[Equation] [Equation]
(1 − β
[Equation]
) (cid:80)
[Equation]
τ =
[Equation]
τ 1 g (
[Equation] [Equation]
i
[Equation]
(cid:15) (
[Equation]
i (cid:114) (
[Equation]
(cid:80) Hτ =
[Equation]
(cid:16) g (
[Equation]
(cid:17)
[Equation]
+ (cid:15) (
[Equation]
+ ξ , (32) where ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 1 n [ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t )
d
[ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 2 n,i (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 2 d,i (cid:12)(cid:12)(cid:12) ≤ ˜ O ( ηξ 2 ) .
Let’s ﬁrst look at the update of W ( t )
[ i, j ] .
For t in the ﬁrst phase
[Equation]
we write the RHS of eq.
(32) as
[Equation]
where
[Equation]
We have already shown that (cid:12)(cid:12)(cid:12)
[Equation]
( t ) 1 n
[Equation]
i, j ]
[Equation] [Equation]
(cid:15) ( t )
d [ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O ( ηξ 2 ) .
By Lemma 19, we have that ∀ i, j ∈ [ d ] ,
[Equation]
g ( t )
[ i, j ] (cid:12)(cid:12)(cid:12)
[Equation]
O ( √ η ) .
Similarly, we have ∀ i, j ∈ [ d ] ,
[Equation] [Equation]
[Equation]
By Lemma 18, we know that (cid:12)(cid:12)(cid:12) g ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
Ω (cid:0) √ η (cid:1) .
Then we have that ∀ i, j ∈ [ d ] : (cid:12) (cid:12)(cid:12) (cid:15) ( t ) 1 n [ i, j ] (cid:12) (cid:12)(cid:12) ≤ ˜ O ( ηξ 2 ) ≤ ˜ O ( √ η ) (cid:12) (cid:12)(cid:12) g ( t )
[ i, j ] (cid:12) (cid:12)(cid:12) ,
[Equation]
(cid:12)(cid:12) (cid:15) ( t )
d
[ i, j ] (cid:12) (cid:12)(cid:12) ≤ ˜ O ( √ η ) ξ 2 .
Therefore by Lemma 33 in Appendix G, we have
[Equation]
( t ) 1 n [ i, j ] (cid:114)(cid:16) g ( t )
[ i, j ] (cid:17) 2 (cid:0) 1 − β H +1(cid:1)
+ e ( t )
d
[ i, j ] + (cid:15) ( t )
d [ i, j ] + ξ = 1 − β H +1(cid:113) 1 − β H +1(cid:16) sign (cid:16) g ( t )
[ i, j ] (cid:17) + ˜ e ( t )
[ i, j ] (cid:17) , where (cid:12) (cid:12)(cid:12)
˜
e ( t )
[ i, j ] (cid:12) (cid:12)(cid:12) =
˜ O (cid:0) √ η (cid:1) .
Since β ∈ (0 , 1) , we know that log β ≤ β − 1 < 0 .
Then our choice of H gives us H = 1 1 − β 1 log dηξ 2 ≥ log ηξ
2 d log β 1 and H > 1 1 − β 2 log dηξ 2 ≥ log ηξ 2 d log β 2 , which implies that β H 1 , β H 2 ≤ ηξ 2 /d .
Hence for t ≥ t inc > H , η t 1 − β H +1 1 √ 1 − β H +1=
η √ 1 − β t +12 √ 1 − β H +11 − β H +11 − β t +1= η
(1 ± O ( η )) .
Combining
[Equation]
, where (cid:12)(cid:12)(cid:12) e ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:0) √ η (cid:1) .
The proof for w ( t ) 2 i is similar.
So far we have successfully proved eq.
By sign (cid:16) ∆ W ( t ) 1 [ i, j ] (cid:17) = sign (cid:16) ∆ w
[Equation]
t
[Equation]
2 i (cid:17) = sign (cid:16) w (
[Equation]
)2 i (cid:17) in Lemma 18, we know that sign (cid:16) − g ( t ) 1 [ i, j ] (cid:17) = sign (cid:16) − g ( t )
i (cid:17) = sign (cid:16) w (0)2 i (cid:17) , which gives us ∀ i, j ∈ [ d ] : w ( t )
i = sign (cid:16) w (0)2 i (cid:17) η
( t − t inc ) + R ( t )
i ,
W ( t ) 1 [ i, j ] = sign (cid:16) w (0)2 i (cid:17) η ( t − t inc ) + R ( t ) 1 [ i, j ] , where (cid:12)(cid:12)(cid:12) R ( t )
[ i,j ] (cid:12)(cid:12)(cid:12) η ( t − t inc ) =
˜ O (cid:18)
√ η + (cid:12)(cid:12)(cid:12) W ( t inc )
[ i,j ] (cid:12)(cid:12)(cid:12) η ( t − t inc ) (cid:19) and (cid:12)(cid:12)(cid:12) R ( t )
i (cid:12)(cid:12)(cid:12) η ( t − t inc ) =
˜ O (cid:18)
√ η + (cid:12)(cid:12)(cid:12) w ( t inc ) 2 i (cid:12)(cid:12)(cid:12) η ( t − t inc ) (cid:19) .
Now it sufﬁces to show that ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) w ( t inc ) 2 i (cid:12)(cid:12)(cid:12) ≤ O (cid:0) 1 d α (cid:1) , (cid:12)(cid:12)(cid:12) W ( t inc )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ O (cid:0)
d α (cid:1) , which is implied by Lemma 17.
Finally to complete the proof, we show that T 1 = Θ (cid:16) 1 √ dη (cid:17) .
When t = T 1 , we have ∀ j ∈
[ d ] :
[Equation]
[ i, j ] = Θ(1) .
Combining with the above results, we know that dη 2 ( T 1 − t inc ) 2 = Θ(1) , i.e. η ( T 1 − t inc ) =
Θ (cid:16) 1 √ d (cid:17) .
In Section D.5, we will prove t inc = Θ (cid:18) ηd
α + (cid:19) .
Then we have T 1 = Θ (cid:16) 1 √ dη (cid:17) .
D. P ROOF OF L EMMA
For certain t and H , we write eq.
(27) as
[Equation]
where
[Equation]
and r ( t ) 1 n [ i, j ] , r ( t )
d [ i, j ] , r ( t ) 2 n,i , r ( t ) 2 d,i are deﬁned in eq.
Since β 2 = β 21 < β 1 , then if we pick H ≥
[Equation]
( t ) 1 ,G
[Equation]
t
[Equation] [Equation]
G
[Equation]
t )
[Equation] [Equation]
G
[Equation]
t )
[Equation]
ηξ
[Equation]
, we can get that
[Table]
[Equation]
ηξ 2 .
Hence we can apply Lemma 32 in Appendix G to get that (cid:12)(cid:12)(cid:12) q ( t
[Equation]
n [ i, j ]
[Equation] [Equation]
[ i, j ]
[Equation] [Equation]
[Table]
[Equation]
Pick T in Lemma 31 as of order ˜ O (cid:16) 1 √ dη (cid:17) .
By Lemma 31, we have with probability at least 1 − 1 d , for all t ≤ T , ∀ τ ≤ t and ∀ i, j ∈ [ d ] ,
[Equation]
[Equation]
Plugging into eq.
(28) gives us
[Equation] [Equation]
j ]
[Equation]
Dg ( τ )
[ i, j ] (cid:12) (cid:12)(cid:12)≤ O (cid:18)
D ( t ) 1 G ( t ) 1 + (cid:16) D ( t ) 1 (cid:17) 2 (cid:19) , (cid:12)(cid:12)(cid:12) r ( t ) 2 n,i (cid:12)(cid:12)(cid:12) ≤ (1 − β 1 ) t (cid:88) τ
= β t − τ(cid:12)(cid:12)(cid:12)
Dg ( τ ) 2 i (cid:12)(cid:12)(cid:12) ≤ O (cid:16) D ( t ) 2 (cid:17) , (cid:12)(cid:12)(cid:12) r ( t ) 2 d,i (cid:12)(cid:12)(cid:12) ≤ (1 − β 2 ) t (cid:88) τ
= β t − τ(cid:12)(cid:12)(cid:12) 2 g ( τ )
i Dg ( τ ) 2 i (cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12) Dg ( τ ) 2 i (cid:12)(cid:12)(cid:12) 2 ≤ O (cid:18) D ( t ) 2 G ( t ) 2 + (cid:16) D ( t ) 2 (cid:17) 2 (cid:19) .
D. P ROOF OF C OROLLARY
Since G ( t ) 1 ≤ ˜ O (cid:16) 1 √ d (cid:17) , G ( t ) 2 ≤ ˜ O (cid:16) √ d (cid:17) , then H := 1 1 − β 1 log dηξ 2 is bigger than 1 1 − β 1 log max (cid:26) G ( t ) 1 ,G ( t ) 2 , (cid:16) G ( t ) 1 (cid:17) 2 , (cid:16) G ( t ) 2 (cid:17) 2 (cid:27) ηξ.
By M ( t ) 1 , M ( t ) 2 ≤ ˜ O (cid:16) 1 √ d (cid:17) , G ( t ) 1 ≤ ˜ O (cid:16) 1 √ d (cid:17) , G ( t ) 2 ≤ ˜ O (cid:16) √ d (cid:17) and the assumption σ ≤ η 3 / 2 ξ 2 d 13 / 4 , we get that D ( t ) 1 and D ( t ) 2 are upper bounded by D ( t ) 1 ≤ ˜ O
(cid:0)
d 7 / 4 ση − 1 / 2 (cid:1) and D ( t )
≤ ˜ O (cid:0) d 11 / 4 ση − 1 / 2 (cid:1) , which yields ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 1 n [ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t )
d
[ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 2 n,i (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 2 d,i (cid:12)(cid:12)(cid:12) ≤ ˜ O ( ηξ 2 ) .
D. P ROOF OF L EMMA
The proof is based on the following two lemmas.
Lemma 20.
Under Assumption 1 and 2, we have that w.p. at least 1 − 1 d α 2 − 1 , for every 1 ≤ i ≤ d , √ π d 32 α ≤ (cid:12)(cid:12)(cid:12) w (0)2 i (cid:12)(cid:12)(cid:12) ≤ (cid:113) 2 d 2 α log 2 dδ , and that w.p. at least 1 − δ for any given δ > 0 , (cid:12)(cid:12)(cid:12) W (0)
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ (cid:113)
d 4 α log 2 d 2 δ .
Lemma 21.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
Pick β 2 = β 21 , ξ ∈ (0 , 1) , η < 14 .
Consider any time point t ≤ ˜ O (cid:16) 1 √ dη (cid:17) .
If ∀ τ ≤ t, ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) W ( τ ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) 1 √ d (cid:17) , (cid:12)(cid:12)(cid:12) w ( τ )
i (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) 1 √ d (cid:17) and (cid:12) (cid:12)(cid:12) g ( τ ) 1 [ i, j ] (cid:12) (cid:12)(cid:12) ≤ ˜ O (cid:16) 1 √ d (cid:17) , (cid:12) (cid:12)(cid:12) g ( τ ) 2 i (cid:12) (cid:12)(cid:12) ≤ ˜ O (cid:16) √ d (cid:17) , we will have
[Equation]
, where the ˜ O notation depends on H =
[Equation]
Furthermore, if for certain i, j ∈ [ d ] , Condition 1 (resp.
Condition 2) is satisﬁed, we will have sign (cid:16) ∆ W ( t ) 1 [ i, j ] (cid:17) =
− s ( t )
[ i, j ] ,
[Equation]
∆ W ( t ) 1 [ i,
[Equation] [Equation]
= ˜Θ( η ) (cid:16) resp.
sign
[Equation]
w
[Equation]
t )
i
[Equation]
= − s
[Equation]
t ) 2 i ,
[Equation]
w
[Equation]
t )
i (cid:12)(cid:12)(cid:12) = ˜Θ( η ) (cid:17) .
Now we prove Lemma 17.
Deﬁne t d := inf (cid:110) t : ∃ i, j : (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) > 1 d or (cid:12)(cid:12)(cid:12) w ( t )
i (cid:12)(cid:12)(cid:12) > 1 d (cid:111) .
Now we want to ﬁnd a time point t inc before t d for the lemma to hold.
During the period t < t d , we have ∀ j ∈
[ d ] , E j = − Θ(1) (which means t d < T 1 ) and therefore for all i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) g ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ 1 d and (cid:12)(cid:12)(cid:12) g ( t )
i (cid:12)(cid:12)(cid:12) ≤ 1 .
Then we can use Lemma 21 to get that for t ≤ min (cid:110) t d , 1 √ dη (cid:111) , we have (cid:12)(cid:12)(cid:12) ∆ W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O ( η ) , (cid:12)(cid:12)(cid:12) ∆ w ( t ) 2 i (cid:12)(cid:12)(cid:12) ≤ ˜ O ( η ) .
Hence t d ≥ ˜Ω (cid:16) 1 ηd (cid:17) .
Deﬁne t sign = inf (cid:110) t < min (cid:110) t d , 1 √ dη (cid:111) :
∃ i ∈ [ d ] : (cid:12)(cid:12)(cid:12) w ( t ) 2 i (cid:12)(cid:12)(cid:12) ≤ 1 d 32 α (cid:111) .
By Lemma 20, w.h.p.
∀ i ∈
[ d ] : (cid:12)(cid:12)(cid:12) w (0)2 i (cid:12)(cid:12)(cid:12) ≥ √ π d 32 α , combining with (cid:12)(cid:12)(cid:12) ∆ w ( t ) 2 i (cid:12)(cid:12)(cid:12) ≤ ˜ O ( η ) gives us that w.h.p.
, t sign ≥ √ π − 1 d 32 α / ˜ O ( η ) = ˜Ω (cid:18)ηd 32 α (cid:19) .
Now let’s analyze the behavior of W 1 during the period t < t sign .
Consider any i, j ∈ [ d ] .
By deﬁnition, sign (cid:16) w ( t )
i (cid:17) = sign (cid:16) w (0)2 i (cid:17) .
Note that E ( t ) j = − Θ(1) , then we have sign (cid:16) g ( t )
[ i, j ] (cid:17) =
− sign (cid:16) w (0)2 i (cid:17) and that (cid:12)(cid:12)(cid:12) g ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
Ω (cid:16) 1 d 32 α (cid:17) =
Ω( ξ ) by our choice of ξ .
Then we know that
Condition 1 is satisﬁed with s ( t )
[ i, j ] = − sign (cid:16) w (0)2 i (cid:17) (for all H < t ≤ t sign ), which by Lemmayields sign (cid:16) ∆ W ( t )
[ i, j ] (cid:17)
= sign (cid:16) w (0)2 i (cid:17) and (cid:12)(cid:12)(cid:12) ∆ W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) = ˜Θ( η ) .
Lemma 20 tells us that w.h.p.
, ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) W (0) 1 [ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:0) 1 d 2 α (cid:1) .
For any i, j , if initially sign (cid:16) W (0)1 [ i, j ] (cid:17) =
sign (cid:16) w (0)2 i (cid:17) , then for the following steps before t sign , we will have sign (cid:16) W ( t ) 1 [ i, j ] (cid:17)
= sign (cid:16) w (0)2 i (cid:17) .
If initially sign (cid:16) W (0)1 [ i, j ] (cid:17) (cid:54) = sign (cid:16) w (0)2 i (cid:17) , then after at most t 0 = ˜ O (cid:16) 1 ηd 2 α (cid:17) steps, W 1 [ i, j ] will ﬂip the sign.
Note that t
= ˜ O (cid:16) 1 ηd 2 α (cid:17) is smaller than t sign .
Hence we have shown that at some time point t 0 , we have ∀ i, j ∈ [ d ] : sign (cid:16) W ( t ) 1 [ i, j ] (cid:17) = sign (cid:16) w ( t )
i (cid:17) = sign (cid:16) w (0)2 i (cid:17) .
Now we analyze the period t ≥ t 0 .
When t 0 < t ≤ t sign , we still have sign (cid:16) ∆ W ( t ) 1 [ i, j ] (cid:17)
= sign (cid:16) w (0)2 i (cid:17) and (cid:12)(cid:12)(cid:12) ∆ W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) = ˜Θ( η ) .
Combining these two with the fact sign (cid:16) W ( t 0 )[ i, j ] (cid:17)
= sign (cid:16) w (0)2 i (cid:17) , we know that for all t ∈ [ t 0 , t sign ] , sign (cid:16) W ( t )
[ i, j ] (cid:17)
= sign (cid:16) w (0)2 i (cid:17) and that ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) W ( t +1)
[ i, j ] (cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) + ˜Θ( η ) .
Then at certain step t inc which satisﬁes t inc = t 0 + ˜Θ (cid:18) ηd
α + (cid:19) ∈
( H, t sign ) , we will have ∀ t inc − H ≤ τ ≤ t inc , ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) W ( τ )
[ i, j ] (cid:12)(cid:12)(cid:12) =
Θ (cid:16) d 32 α + (cid:17) and therefore (cid:12)(cid:12)(cid:12) g ( τ ) 2 i (cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12)(cid:80) dj
=1 W ( τ )
[ i, j ] E ( τ ) j (cid:12)(cid:12)(cid:12) =
(cid:80) dj
=1 (cid:12)(cid:12)(cid:12) W ( τ )
[ i, j ] E ( τ ) j (cid:12)(cid:12)(cid:12) =
Θ (cid:16) 1 d 32 α (cid:17) =
Ω( ξ ) .
For t ≤ t inc , we have ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
O (cid:16) d 32 α + (cid:17) .
Since t inc < t sign , we have (cid:12) (cid:12) w t inc 2 i (cid:12) (cid:12) = Ω (cid:16) 1 d 32 α (cid:17) .
For t ≤ t inc , note that (cid:12) (cid:12) (cid:12) ∆ w ( t ) 2 i (cid:12) (cid:12) (cid:12) ≤ ˜ O ( η ) , t inc = t 0 + ˜Θ (cid:18) ηd
32 α + (cid:19) =
˜Θ (cid:18) ηd 32 α + (cid:19) , combining with the upper bound in Lemma 20 yields (cid:12)(cid:12)(cid:12) w ( t )
i (cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12) w (0)2 i
[Equation]
+
[Equation]
inc
[Equation] [Equation]
Moreover, ∀ t inc − H ≤ τ ≤ t inc , ∀ i ∈ [ d ] : sign (cid:16) g ( τ ) 2 i (cid:17) = − sign (cid:16) w (0)2 i (cid:17) .
Then Condition 2 is satisﬁed with s ( t ) 2 i = − sign (cid:16) w (0)2 i (cid:17) for t = t inc .
In the analysis of g ( t )
[ i, j ] , we have already shown that for all t ≤ t sign (and thus for t = t inc ), Condition 1 is satisﬁed, which completes the proof.
D. P ROOF OF L EMMA
Since for X ∼ N (cid:0) 0 , σ 2 (cid:1) , we have that P ( | X | ≤ t ) ≤ 2 t √ 2 πσ , then for a ﬁxed i ,
[Equation]
Then by union bound, we have that w.p. at least 1 −
[Equation]
1 , for every 1 ≤ i ≤ d
[Equation] [Equation]
As for the upper bounds, using the Gaussian tail bound and union bound, we have w.p. at least 1 − δ , ∀ i, j ∈ [ d ] :
[Equation]
(cid:12) ≤ (cid:114)d 2 α log 2 d δ , (cid:12)(cid:12) (cid:12) W (0)1 [ i, j ] (cid:12)(cid:12) (cid:12) ≤ (cid:114)d 4 α log 2 dδ .
D. P ROOF OF L EMMA
Now we analyze the magnitude order of ∆ W ( t ) 1 [ i, j ] .
The analysis of ∆ w ( t ) 2 i is similar.
For t ≤ ˜ O (cid:16) 1 √ dη (cid:17) .
By assumption, M ( t ) 1 , M ( t ) 2 ≤ ˜ O (cid:16) 1 √ d (cid:17) , G ( t ) 1 ≤ ˜ O (cid:16) 1 √ d (cid:17) , G ( t ) 2 ≤ ˜ O (cid:16) √ d (cid:17) , and σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
Hence we can pick H :
= 1 1 − β 1 log dηξ 2 and apply Lemma 16 and Corollary 2 to get that, w.h.p., for all t ≤ ˜ O (cid:16) 1 √ dη (cid:17) and
∀ i, j ∈ [ d ] , eq. (27) can be written as
[Equation]
where ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 1 n [ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t )
d [ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 2 n,i (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 2 d,i (cid:12)(cid:12)(cid:12) ≤ ˜ O ( ηξ 2 ) .
On one hand, using (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 1 n [ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 1 d [ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O ( ηξ 2 ) and β 2 = β 21 , and √ x + y ≥ √ x − (cid:112) | y | when x ≥ 0 , x + y ≥ 0 , we get from eq.
(33) that (cid:12)(cid:12)(cid:12) ∆ W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ η t (1 − β 1 ) (cid:12)(cid:12)(cid:12)(cid:80)
Hτ =
β τ 1 g ( t − τ )
[ i, j ] (cid:12)(cid:12)(cid:12) + ˜ O ( ηξ 2 ) (cid:114) (1 − β 2 ) (cid:80)
Hτ =0 (cid:16) β τ 1 g ( t − τ )
[Equation]
√
[Equation]
√
[Equation] [Equation]
where ( i ) uses Cauchy-Schwarz inequality for the numerator.
On the other hand, when sign (cid:16) g ( t − H )
[ i, j ] (cid:17)
= sign (cid:16) g ( t − H
+
[ i, j ] (cid:17) = ...
= sign (cid:16) g ( t )
[ i, j ] (cid:17) =
s ( t )
[ i, j ] , we have sign (cid:32) H (cid:88) τ
= β τ 1 g ( t − τ )
[ i, j ] (cid:33) =
s ( t )
[ i, j ] , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
H (cid:88) τ = β τ 1 g ( t − τ )
[ i, j ] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≥ (cid:118)(cid:117)(cid:117)(cid:116)
H (cid:88) τ
= (cid:16) β τ 1 g ( t − τ )
[ i, j ] (cid:17) 2 .
If we further have (1 − β 1 ) (cid:12)(cid:12)(cid:12)(cid:80) Hτ =0 β ( τ ) 1 g ( t − τ )[ i, j ] (cid:12)(cid:12)(cid:12) ≥ Ω( ξ ) , then combining with (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 1 n [ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O ( ηξ 2 ) <
ξ we will get sign (cid:16) ∆ W ( t ) 1 [ i, j ] (cid:17) =
− sign (cid:32) H (cid:88) τ = β τ 1 g ( t − τ )
[ i, j ] + (cid:15) ( t )
n
[ i, j ] (cid:33) =
− sign (cid:32) H (cid:88) τ = β τ 1 g ( t − τ )
[ i, j ] (cid:33) =
− s ( t )
[ i, j ] .
Using √ x + y ≤ (cid:112) | x | + (cid:112) | y | , we obtain that
[Equation]
[ i
[Equation]
j ] (cid:12)(cid:12)(cid:12)
[Equation]
(cid:114) (
[Equation]
g
[Equation]
[ i, j ]
[Equation]
O
[Equation]
≥ η t
[Equation]
(cid:12)(cid:12)(cid:12)(cid:80)
Hτ =
β ( τ ) 1 g ( t − τ )
[ i, j ] (cid:12)(cid:12)(cid:12)
[Equation]
Together with the upper bound completes the proof.
D. P ROOF OF L EMMA
The proof is based on the following lemma, which gives a coarse analysis on the magnitude of weights and their increments per step during the ﬁrst phase.
Lemma 22.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
Pick ξ ≤ min (cid:110)(cid:112) η d 3 α − 1 , 1 d 32 α (cid:111) , for t inc in Lemma
[Equation]
we have that w
[Equation]
h
[Equation]
p. for all t inc ≤ t ≤ T 1 , ∀ i, j ∈ [ d ] .
[Equation]
[Equation]
w ( t ) 2 i (cid:12)(cid:12)(cid:12) =
˜ O (cid:18) 1 √ d (cid:19) .
Specially, at the end of the ﬁrst phase ( t = T 1 ), we have ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) w ( T 1 ) 2 i (cid:12)(cid:12)(cid:12) = ˜Θ (cid:16) 1 √ d (cid:17) and (cid:12)(cid:12)(cid:12) W ( T 1 )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ Θ (cid:16) 1 √ d (cid:17) .
Now we go back to the proof of Lemma 18.
For t inc ≤ t < T 1 , since E ( t ) j = (cid:16) W ( t ) 2 W ( t ) 1 (cid:17) j −
A j = (cid:80) di
=1 w ( t )
i W ( t )
[ i, j ] − A j , we have, ∆ E (
[Equation] [Equation]
:=
[Equation] [Equation]
Combining Lemma 22 and eq.
(34) gives us ∀ j ∈ [ d ] ,
[Equation]
( t )
[Equation]
(cid:12)(cid:12)(cid:12) =
d (cid:88) i =
[Equation]
i
[Equation]
≤
[Equation]
i =
[Equation]
O (cid:18)
[Equation]
√ d (cid:19) =
˜ O (cid:16)
[Equation]
Let’s ﬁrst analyze g ( t ) 1 [ i, j ] .
Note that
[Equation]
where sign (cid:16) w ( t +1)
i ∆ E ( t ) j (cid:17)
= sign (cid:16) w (0)2 i (cid:17) while sign (cid:16) ∆ w ( t )
i E ( t ) j (cid:17)
= − sign (cid:16) w (0)2 i (cid:17) .
Now we analyze the sign of g ( t )
[ i, j ] when t inc ≤ t < T 1 .
Using (cid:12)(cid:12)(cid:12) w ( t inc +1) 2 i (cid:12)(cid:12)(cid:12) =
˜ O (cid:0) 1 d α (cid:1) and eq.
(35), we get that (cid:12)(cid:12)(cid:12) w ( t inc +1) 2 i ∆ E ( t inc ) j (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) 1 d α · √ dη (cid:17) .
While on the other hand, (cid:12)(cid:12)(cid:12) ∆ w ( t inc ) 2 i E ( t inc ) j (cid:12)(cid:12)(cid:12) = ˜Θ( η ) .
That means sign (cid:16) ∆ g ( t inc )[ i, j ] (cid:17)
= − sign (cid:16) w (0)2 i (cid:17) .
Note that sign (cid:16) g ( t inc )[ i, j ] (cid:17) = − sign (cid:16) w ( t inc ) 2 i (cid:17) =
− sign (cid:16) w (0)2 i (cid:17) , we know that (cid:12)(cid:12)(cid:12) g ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) will increase when t = t inc .
In the following steps, (cid:12)(cid:12)(cid:12) g ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) will keep increasing as long as (cid:12)(cid:12)(cid:12) ∆ w ( t )
i E ( t ) j (cid:12)(cid:12)(cid:12) > (cid:12)(cid:12)(cid:12) w ( t +1) 2 i ∆ E ( t ) j (cid:12)(cid:12)(cid:12) .
Since (cid:12)(cid:12)(cid:12) W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) w ( t ) 2 i (cid:12)(cid:12)(cid:12) keep increasing while (cid:12)(cid:12)(cid:12) ∆ W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) ∆ w ( t ) 2 i (cid:12)(cid:12)(cid:12) remain ˜Θ( η ) , by eq.
(35), we know that the trend of (cid:12)(cid:12)(cid:12) ∆ E ( t ) j (cid:12)(cid:12)(cid:12) is to increase.
On the other hand, (cid:12)(cid:12)(cid:12) E ( t ) j (cid:12)(cid:12)(cid:12) keeps decreasing since
E ( t )
j < 0 while ∆ E ( t ) j > 0 .
Then after some time point we will have (cid:12)(cid:12)(cid:12) ∆ w ( t ) 2 i E ( t ) j (cid:12)(cid:12)(cid:12) < (cid:12)(cid:12)(cid:12) w ( t +1) 2 i ∆ E ( t ) j (cid:12)(cid:12)(cid:12) and in the following steps (cid:12) (cid:12)(cid:12) g ( t )
[ i, j ] (cid:12) (cid:12)(cid:12) will have the trend to decrease.
Specially, when t = T 1 − 1 , we have (cid:12)(cid:12)(cid:12) E ( t ) j (cid:12)(cid:12)(cid:12) =
Θ (cid:0) √ ηd (cid:1) and (cid:12)(cid:12)(cid:12) W ( t ) 1 (cid:12)(cid:12)(cid:12) = ˜Θ (cid:16) 1 √ d (cid:17) , (cid:12)(cid:12)(cid:12) w ( t +1) 2 i (cid:12)(cid:12)(cid:12) = ˜Θ (cid:16) 1 √ d (cid:17) by Lemma 22, which gives us (cid:12)(cid:12)(cid:12) ∆ E ( t ) j (cid:12)(cid:12)(cid:12) =
d (cid:88)
i = (cid:12)(cid:12)(cid:12) w ( t +1)
i ∆ W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12) ∆ w ( t ) 2 i W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ d (cid:88)
i = ˜Θ (cid:18) η 1 √ d (cid:19) =
˜Θ (cid:16) η √ d (cid:17) .
Hence (cid:12)(cid:12)(cid:12) w ( t +1) 2 i ∆ E ( t ) j (cid:12)(cid:12)(cid:12) = ˜Θ( η ) > (cid:12)(cid:12)(cid:12) ∆ w ( t )
i E ( t ) j (cid:12)(cid:12)(cid:12) = ˜Θ (cid:0) η √ ηd (cid:1) .
Therefore we have proved that when t inc ≤ t < T 1 , the trend of (cid:12)(cid:12)(cid:12) g ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) is to ﬁrst increase and then decrease.
In order to prove (cid:12) (cid:12) (cid:12) g ( t )
[ i, j ] (cid:12) (cid:12) (cid:12) = ˜Ω (cid:0) √ η (cid:1) , it sufﬁces to show that (cid:12) (cid:12) (cid:12) g ( t inc )[ i, j ] (cid:12) (cid:12) (cid:12) = ˜Ω (cid:0) √ η (cid:1) and (cid:12)(cid:12)(cid:12) g ( T 1 )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜Ω (cid:0) √ η (cid:1) .
When t = t inc , (cid:12)(cid:12)(cid:12) g ( t inc )[ i, j ] (cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12) w ( t inc ) 2 i (cid:12)(cid:12)(cid:12) · (cid:12)(cid:12)(cid:12) E ( t inc ) j
(cid:12)(cid:12)(cid:12)
[Equation]
When t = T 1 , we have
[Equation]
As for g ( t ) 2 i , since for ∀ i ∈ [ d ] , W ( t ) 1 [ i, j ] for different j have the same sign.
Combining with ∀ j ∈ [ d ] : E ( t )
j < 0 gives us
[Equation]
Then it sufﬁces to show that for t inc ≤ t < T 1 , (cid:12)(cid:12)(cid:12) E ( t ) j W ( t )
[ i, j ]
[Equation]
= ˜Ω (cid:0) √ η (cid:1) , which can be proven using the same technique as above.
Finally, for ∀ τ ≤ t, ∀ i, j ∈ [ d ] , note that the upper bounds of (cid:12)(cid:12)(cid:12) W ( τ ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12) w ( τ )
i (cid:12)(cid:12)(cid:12) are already given in Lemma 22.
As for (cid:12)(cid:12) (cid:12) g ( τ )
[ i, j ] (cid:12)(cid:12) (cid:12) and (cid:12)(cid:12) (cid:12) g
( τ )
i (cid:12)(cid:12) (cid:12) , we have (cid:12)(cid:12) (cid:12) g ( τ ) 1 [ i, j ] (cid:12)(cid:12) (cid:12) = (cid:12)(cid:12) (cid:12) w ( τ ) 2 i E ( τ ) j (cid:12)(cid:12) (cid:12) =
˜ O (cid:16) 1 √ d (cid:17) , (cid:12)(cid:12) (cid:12) g
( τ )
i (cid:12)(cid:12) (cid:12) ≤
[Table]
W
[Table]
[ i,
[Table]
D. P ROOF OF L EMMA
For any i, j ∈ [ d ] , and any t in the interval [ t inc , T 1 ] , we prove by induction that
[Equation]
t )
i (cid:12)(cid:12)(cid:12) =
˜ O (cid:16) 1 √ d (cid:17) .
(B) ∀ τ ∈ [ t − H, t ] : sign (cid:16) W ( τ )
[ i, j ] (cid:17)
= sign (cid:16) w ( τ ) 2 i (cid:17)
= sign (cid:16) w (0)2 i (cid:17) .
(C) (cid:12)(cid:12)(cid:12) g ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) ≥ Ω( ξ ) , (cid:12)(cid:12)(cid:12) g ( t )
i (cid:12)(cid:12)(cid:12) ≥ Ω( ξ ) .
The base case t = t inc was already proven by Lemma 17.
For t ∈ [ t inc , T 1 ) , suppose (B) and (C) hold for time t and (A) holds for all τ ∈ [ t inc , t ] .
From (A), we get that ∀ τ ∈ [ t inc , t ] : (cid:12)(cid:12)(cid:12) g ( τ ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12) w ( τ ) 2 i E ( τ ) j (cid:12)(cid:12)(cid:12) =
˜ O (cid:16) 1 √ d (cid:17) , (cid:12)(cid:12)(cid:12) g
( τ ) 2 i (cid:12)(cid:12)(cid:12) ≤ (cid:80) dj
=1 (cid:12)(cid:12)(cid:12) E ( τ ) j W ( τ )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:16) √ d (cid:17) .
Since when t < T 1 , ∀ j ∈ [ d ] : E ( t ) j < 0 , from (B) we know that ∀ τ ∈ [ t − H, t ] : sign (cid:16) g ( τ ) 1 [ i, j ] (cid:17) = sign (cid:16) g
( τ )
i (cid:17) = − sign (cid:16) w (0)2 i (cid:17) .
Combining with (C) tells us that Condition 1 and 2 are satisﬁed.
In Section D.2 we have shown that T 1 = Θ (cid:16) 1 √ dη (cid:17) .
Then for t ∈ [ t inc , T 1 ) , we can use Lemma 21 to get that ∀ t inc ≤ τ ≤ t , ∀ i, j ∈ [ d ] , sign (cid:16) ∆ W
[Equation]
τ )
[ i, j ] (cid:17)
= sign (cid:16) ∆ w (
[Equation] [Equation]
i (cid:17) = sign
[Equation]
w (0)2 i (cid:17) , (cid:12)(cid:12)(cid:12)
[Equation]
=
[Equation] [Equation]
w
[Equation]
i
[Equation]
= ˜Θ( η ) .
Since when t = t inc , sign (cid:16) W ( t inc )[ i, j ] (cid:17) = sign (cid:16) w ( t inc ) 2 i (cid:17) = sign (cid:16) w (0)2 i (cid:17) .
We get that for t inc ≤ τ ≤ t , ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) W ( τ +1)[
[Equation] [Equation]
+
[Equation]
) , (cid:12)(cid:12)(cid:12) w ( τ +1) 2 i (cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12) w ( τ )
i (cid:12)(cid:12)(cid:12) + ˜Θ( η ) .
Now for t + 1 , we have ∀ i, j ∈ [ d ] , sign (cid:16) W ( t +1)
[ i, j ] (cid:17)
=
sign (cid:16) w (0)2 i (cid:17) , (cid:12) (cid:12)(cid:12) W ( t +1)
[ i, j ] (cid:12) (cid:12)(cid:12) =
(cid:12) (cid:12)(cid:12) W
( t inc )[ i, j ] (cid:12) (cid:12)(cid:12)
+ ( t + 1 − t inc ) ˜Θ( η ) , sign (cid:16) w ( t +1)
i (cid:17) = sign (cid:16) w (0)2 i (cid:17) , (cid:12)(cid:12)(cid:12) w ( t +1) 2 i (cid:12)(cid:12)(cid:12) = (cid:12)(cid:12)(cid:12) w ( t inc ) 2 i (cid:12)(cid:12)(cid:12) + ( t + 1 − t inc ) ˜Θ( η ) .
That means ∀ τ ∈
[ t + 1 − H, t + 1] : sign (cid:16) W ( τ )
[ i, j ] (cid:17)
= sign (cid:16) w ( τ ) 2 i (cid:17)
= sign (cid:16) w (0)2 i (cid:17) .
This proves (B) for time t + 1 .
[Table]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
d (cid:88)
j = (cid:12)(cid:12)(cid:12) E ( t +1)
j
W ( t +1)
[ i, j ] (cid:12)(cid:12)(cid:12) ≥ d Θ (cid:18)
d 32 α + (cid:19) (cid:112) ηd
= Ω( ξ ) .
This proves (C) at time t + 1 .
Since t + 1 ≤ T 1 which means ∀ j ∈
[ d ] : (cid:16) W ( t +1)
W ( t +1)(cid:17)
j ≤ O (1) , we obtain that d (cid:88) i = w ( t +1)
i
W ( t +1)
[ i, j ] =
d (cid:88) i = (cid:12)(cid:12)(cid:12) w ( t +1)
2 i (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) W ( t +1)
[ i, j ] (cid:12)(cid:12)(cid:12) =
d (cid:88)
i = (cid:16)(cid:12) (cid:12) (cid:12) w ( t inc ) 2 i (cid:12) (cid:12) (cid:12)
+ ( t + 1 − t inc ) ˜ Θ( η ) (cid:17) (cid:16)(cid:12) (cid:12) (cid:12) W ( t inc )
[ i, j ] (cid:12) (cid:12) (cid:12) + ( t + 1 − t inc ) ˜ Θ( η ) (cid:17)
≤ O (1) .
Note that (cid:12)(cid:12)(cid:12) W ( t inc )[ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) w ( t inc ) 2 i (cid:12)(cid:12)(cid:12) < 1 d (since t inc < t d ), we get that ( t + 1 − t inc )˜Θ( η )
=
O (cid:16) 1 √ d (cid:17) , which gives us (cid:12)(cid:12)(cid:12) w ( t +1) 2 i (cid:12)(cid:12)(cid:12) =
˜ O (cid:16) 1 √ d (cid:17) and (cid:12)(cid:12)(cid:12) W ( t +1)
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:16) 1 √ d (cid:17) and hence (A) holds at time t
+ 1 .
Therefore by induction, we can prove that (A), (B), (C) hold for all t inc ≤ t ≤ T 1 .
Then applying
Lemma 21, we get that for all t inc ≤ t ≤ T 1 , ∀ i, j ∈ [ d ] :
[Equation]
= ˜Θ( η ) .
Specially, at the end of the ﬁrst phase, we have ∀ j ∈
[ d ] : (cid:16) W ( t +1)
W ( t +1)(cid:17)
j = Θ(1) .
Repeating the above proof techniques gives us (cid:12)(cid:12)(cid:12) w ( T 1 ) 2 i (cid:12)(cid:12)(cid:12) = ˜Θ (cid:16) 1 √ d (cid:17) and (cid:12)(cid:12)(cid:12) W ( T 1 )
[ i, j ] (cid:12)(cid:12)(cid:12) = ˜Θ (cid:16) 1 √ d (cid:17) for ∀ i, j ∈ [ d ] .
D. P ROOF OF L EMMA
Let’s ﬁrst prove eq.
By Lemma 18, for t inc ≤ t < T 1 , we have ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) g ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜Ω (cid:0) √ η (cid:1) , (cid:12)(cid:12)(cid:12) g ( t ) 2 i (cid:12)(cid:12)(cid:12) = ˜Ω (cid:0) √ ηd (cid:1) .
Then it sufﬁces to show that for t inc ≤ t < T 1 , (cid:12)(cid:12)(cid:12) g ( t )
[ i, j ] − g ( t − τ )
[ i, j ] (cid:12)(cid:12)(cid:12) =
τ ˜ O
( η ) and (cid:12)(cid:12)(cid:12) g ( t ) 2 i − g
( t − τ ) 2 i (cid:12)(cid:12)(cid:12) =
τ
˜ O ( ηd ) .
It sufﬁces to show that when t < T 1 , (cid:12)(cid:12)(cid:12) g ( t +1)
[ i, j ] − g
( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O ( η ) and (cid:12)(cid:12)(cid:12) g ( t +1)
i − g
( t )
i (cid:12) (cid:12)(cid:12) =
˜ O
( ηd ) .
By Lemma 17 and 22, we know that when t < T 1 , ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) ∆ W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O ( η ) , (cid:12)(cid:12)(cid:12) ∆ w ( t ) 2 i (cid:12)(cid:12)(cid:12) ≤ ˜ O ( η ) and that (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) 1 √ d (cid:17) , (cid:12)(cid:12)(cid:12) w ( t )
i (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) 1 √ d (cid:17) .
Then the bound (cid:12)(cid:12)(cid:12) ∆ E ( t ) j (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) η √ d (cid:17) in eq.
(35) hold for all t < T 1 (not only t inc ≤ t < T 1 ).
Substituting these bounds into eq.
(36) gives us ∀ t < T 1 ,
[Equation] [Equation]
η
[Equation]
Similarly, we have that (cid:12)(cid:12)(cid:12) g ( t +1)
i − g
( t )
i (cid:12)(cid:12)(cid:12) =
˜ O
( ηd ) , which proves eq.
Note that for a, b ∈ R :
[Equation]
Then eq.
(31) immediately follows from eq.
D. P ROOF OF L EMMA
We divide Lemma 15 into the following three lemmas.
Combining them together immediately gives us the whole proof.
The ﬁrst lemma below gives us the structure of W 2 in the second phase and that of W 1 under some conditions.
Lemma 23.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
By picking η ≤ O (cid:0) 1 d 3 α (cid:1) , ξ ≤ (cid:112) η d 3 α − 1 , and β 2 = β 21 , we have w.h.p.
for T 1 ≤ t < ˜ T , ∀ i ∈ [ d ] : w ( t +1) 2 i = w ( t )
i − η (cid:16) sign (cid:16) g ( t )
i (cid:17) + e ( t ) 2 i (cid:17) , where (cid:12)(cid:12)(cid:12) e ( t ) 2 i (cid:12)(cid:12)(cid:12) =
˜ O ( √ η ) , and moreover ∀ i ∈
[ d ] :
[Equation]
( t
[Equation]
i = sign (cid:16) w (0)2 i (cid:17) c
( t )
+ R ( t ) 2 i , where (cid:12)(cid:12)(cid:12)
[Equation]
t
[Equation]
i (cid:12)(cid:12)(cid:12)
[Equation]
O (cid:18)
[Equation] [Equation]
(cid:19) .
As for W 1 , if for certain i.j ∈ [ d ] and certain t ∈ [ T 1 , ˜ T ) we have (cid:12)(cid:12)(cid:12) g ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜Ω (cid:0) √ η (cid:1) , then
W ( t +1)
[ i, j ] = W ( t )
[ i, j ] − η (cid:16) sign (cid:16) g ( t )
[ i, j ] (cid:17)
+ e ( t )
[ i, j ] (cid:17) , where (cid:12)(cid:12)(cid:12) e ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O ( √ η ) .
The second lemma below also analyzes the structure of W 1 but removes the conditions in Lemma 23.
Lemma 24.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
By picking η ≤ O (cid:0) 1 d 3 α (cid:1) , ξ ≤ (cid:112) η d 3 α − 1 , and β 2 = β 21 , we have w.h.p.
for T 1 ≤ t < ˜ T , ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) = ˜Ω (cid:16) 1 √ d (cid:17) and for any j ∈ [ d ] ,
W
[Equation]
t
[Equation]
1 [ i, j ] = sign (cid:16) w (0)2 i (cid:17) V ( t )
j + R ( t )
[ i, j ] , where (cid:12)(cid:12)(cid:12)
[Equation]
( t
[Equation]
i,
[Equation]
(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation] [Equation]
(cid:19) .
The third lemma proves the convergence of Adam at time ˜ T .
Lemma 25.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
By picking η ≤ O (cid:0) 1 d 3 α (cid:1) , ξ ≤ (cid:112) η d 3 α − 1 , and β 2 = β 21 , at time ˜ T , we have that w.h.p.
∀ j ∈ [ d ] : (cid:12)(cid:12)(cid:12) E ( ˜ T ) j (cid:12)(cid:12)(cid:12) ≤ ˜ O
(cid:0) d √ ηd (cid:1) , which implies (cid:13) (cid:13)(cid:13)
E ( ˜ T ) (cid:13) (cid:13)(cid:13)2 ≤ ˜ O (cid:0) ηd 4 (cid:1) .
D. P ROOF OF L EMMA
The proof is based on the following lemma, which gives a coarse analysis on the magnitude of weights and their increments per step during the second phase.
Lemma 26.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
By picking η ≤ O (cid:0) 1 d 3 α (cid:1) , ξ ≤ (cid:112) η d 3 α − 1 , and β 2 = β 21 , we have w.h.p.
for all T 1 ≤ t < ˜ T , ∀ i,
[Equation]
[ d ]
[Equation]
Moreover, we have that ∀ i, j ∈ [ d ] :
[Equation]
Equipped with Lemma 26, we are ready to prove Lemma 23.
We will only prove the results of w ( t ) 2 i .
The proof for W ( t )
[ i, j ] uses the same techniques.
Lemma 26 gives us upper bounds of (cid:12)(cid:12)(cid:12) w ( t )
i (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) , as well as (cid:12)(cid:12)(cid:12) ∆ w ( t ) 2 i (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12) ∆ W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) for all i, j ∈
[ d ] .
Then we know that eq.(35) still holds, which gives us ∀ j ∈ [ d ] : (cid:12)(cid:12)(cid:12) E ( t +1)
j − E ( t ) j (cid:12)(cid:12)(cid:12) =
˜ O (cid:16) √ dη (cid:17) .
Then we can use the same strategy in Lemma 19 to prove that (cid:12)(cid:12)(cid:12) g ( t +1)
i − g
( t )
i (cid:12)(cid:12)(cid:12) =
˜ O ( ηd ) .
By deﬁnition, for T 1 ≤ t < ˜ T , we know that (cid:12)(cid:12)(cid:12) g ( t ) 2 i (cid:12)(cid:12)(cid:12) =
Ω (cid:0) d √ η (cid:1) .
Combining with the bound (cid:12)(cid:12)(cid:12) g ( t +1)
i − g
( t )
i (cid:12)(cid:12)(cid:12) =
˜ O
( ηd ) , we know that the g ( t ) 2 i parts in eq.(30) and eq.(31) still hold.
Then we can use the same strategy in Section D.2 to prove that the w ( t ) 2 i part of eq.
(29) still holds, which gives us ∀ i ∈
[Equation]
[Equation]
, where (cid:12)(cid:12)(cid:12) e ( t
[Equation]
(cid:12)(cid:12)(cid:12) =
˜ O ( √ η ) .
By Lemma 14, we have that at the end of the ﬁrst phase ( t = T 1 ), ∀ i ∈ [ d ] : w (
[Equation]
2 i
[Equation]
, where
[Equation]
Combining with ∀ i ∈
[ d ] , ∀ t ≤ T i : sign (cid:16) g ( t )
i (cid:17) = − sign (cid:16) w (0)2 i (cid:17) yields that during the second phase, for t ≤ ˜ T , we have ∀
[Equation] [Equation]
i , where
[Equation] [Equation]
D. P ROOF OF L EMMA
By deﬁnition of T f , there exists j 0 ∈
[ d ] such that E ( τ ) j 0 < −√ ηd for T 1 ≤ t ≤ ˜ T .
We prove by induction that during this period, ∀ i ∈
[ d ] : sign (cid:16) w ( t ) 2 i (cid:17)
= sign (cid:16) W ( t )
[ i, j 0 ] (cid:17)
= sign (cid:16) w (0)2 i (cid:17) and that ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) w ( t )
i (cid:12)(cid:12)(cid:12) = ˜Θ (cid:16) 1 √ d (cid:17) , (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:16) 1 √ d (cid:17) .
The base case ( t = T 1 ) was already proven by Lemma 22.
Now suppose for some t such that
T 1 ≤ t < ˜ T , for all τ such that T 1 ≤ τ ≤ t , we have ∀ i ∈
[ d ] : sign (cid:16) w ( τ ) 2 i (cid:17)
= sign (cid:16) W ( τ )
[ i, j 0 ] (cid:17)
= sign (cid:16) w (0)2 i (cid:17) and that ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) w ( τ ) 2 i (cid:12)(cid:12)(cid:12) = ˜Θ (cid:16) 1 √ d (cid:17) , (cid:12)(cid:12)(cid:12) W ( τ )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:16) 1 √ d (cid:17) .
Using these bounds, we get that ∀ j ∈
[ d ] : (cid:12)(cid:12)(cid:12) E ( τ ) j (cid:12)(cid:12)(cid:12) ≤ (cid:80) di
=1 (cid:12)(cid:12)(cid:12) w ( τ )
i W ( τ )
[ i, j ] (cid:12)(cid:12)(cid:12) + | A j | =
O (1) , which then yields two upper bounds (cid:12)(cid:12) (cid:12) g
( τ )
[ i, j ] (cid:12)(cid:12) (cid:12) = (cid:12)(cid:12) (cid:12) w ( τ ) 2 i E ( τ ) j (cid:12)(cid:12) (cid:12) =
˜ O (cid:16) 1 √ d (cid:17) and (cid:12)(cid:12) (cid:12) g
( τ )
i (cid:12)(cid:12) (cid:12) ≤ (cid:80) dj =1 (cid:12)(cid:12) (cid:12) E ( τ ) j W ( τ ) 1 [ i, j ] (cid:12)(cid:12) (cid:12) =
˜ O (cid:16) √ d (cid:17) .
By deﬁnition of T g , we know that for all T 1 ≤ τ ≤ t , ∀ i ∈
[ d ] : (cid:12)(cid:12)(cid:12) g ( τ ) 2 i (cid:12)(cid:12)(cid:12) ≥ d √ η = Ω( ξ ) and that sign (cid:16) g ( τ )
i (cid:17) = − sign (cid:16) w (0)2 i (cid:17) , which implies that Condition 2 is satisﬁed for ∀ i ∈
[ d ] .
At the end of the proof of this lemma, we will show that ˜ T
= ˜Θ (cid:16) 1 √ dη (cid:17) .
Together with the upper bound of (cid:12)(cid:12)(cid:12) g
( τ )
i (cid:12)(cid:12)(cid:12) , we can apply Lemma 21 to get that w.h.p.
for T 1 ≤ τ ≤ t , sign (cid:16) ∆ w ( τ ) 2 i (cid:17) = sign (cid:16) w (0)2 i (cid:17) and (cid:12)(cid:12)(cid:12) ∆ w ( τ ) 2 i (cid:12)(cid:12)(cid:12) = ˜Θ( η ) .
Combining with the inductive hypothesis sign (cid:16) w ( τ )
i (cid:17) = sign (cid:16) w (0)2 i (cid:17) gives us that (cid:12)(cid:12)(cid:12) w ( τ +1) 2 i (cid:12)(cid:12)(cid:12) = (cid:12)(cid:12)(cid:12) w ( τ ) 2 i (cid:12)(cid:12)(cid:12) + ˜Θ( η ) .
Specially, when τ = t , we get the lower bound (cid:12)(cid:12)(cid:12) w ( t +1) 2 i (cid:12)(cid:12)(cid:12) ≥ (cid:12)(cid:12)(cid:12) w ( t ) 2 i (cid:12)(cid:12)(cid:12) = ˜Ω (cid:16) 1 √ d (cid:17) and that sign (cid:16) w ( t +1)
i (cid:17) = sign (cid:16) w (0)2 i (cid:17) .
Since E ( τ ) j 0 < −√ ηd , we have that ∀
i ∈
[ d ] : (cid:12)(cid:12)(cid:12) g ( τ ) 1 [ i, j 0 ] (cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12) w ( τ )
i (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) E ( τ ) j 0 (cid:12)(cid:12)(cid:12) = ˜Ω (cid:0) √ η (cid:1) =
Ω( ξ ) and that sign (cid:16) g ( τ )
[ i, j 0 ] (cid:17) =
− sign (cid:16) w (0)2 i (cid:17) .
That means Condition 1 is satisﬁed for ∀
i ∈
[ d ] and j 0 .
Using the same technique as when we deal with w ( τ ) 2 i , we get that for T 1 ≤ τ ≤ t ,
∀ i ∈
[ d ] : (cid:12)(cid:12)(cid:12) W ( τ +1)
[ i, j 0 ]
[Equation]
=
[Equation]
[ i, j 0 ] (cid:12)(cid:12)(cid:12) + ˜Θ
[Equation]
η
[Equation]
, sign (cid:16) W ( t +1)
[ i, j 0 ] (cid:17) = sign (cid:16) w (0)2 i (cid:17) and that ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) ∆ W
( τ )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O ( η ) .
Now we analyze the magnitude order of (cid:12)(cid:12)(cid:12) w ( t +1) 2 i (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) W ( t +1)
[ i, j ] (cid:12)(cid:12)(cid:12) .
Let’s ﬁrst analyze (cid:12)(cid:12)(cid:12) w ( t +1)
i (cid:12)(cid:12)(cid:12) .
By Lemma 14, when t = T 1 , ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) w ( T 1 ) 2 i (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) w ( T 1 ) 2 j (cid:12)(cid:12)(cid:12) = 1 ± ˜ O (cid:18) √ η + d α − 1 /(cid:19) , (cid:12)(cid:12)(cid:12) W ( T 1 )[ i, j 0 ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) w ( T 1 ) 2 i (cid:12)(cid:12)(cid:12) = 1 ± ˜ O (cid:18) √ η + d α − 1 /(cid:19) .
Combining with the facts that for T 1 ≤ τ ≤ t , (cid:12)(cid:12)(cid:12) W ( τ +1)
[ i, j 0 ] (cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12) W ( τ )
[ i, j 0 ] (cid:12)(cid:12)(cid:12) + ˜Θ( η ) and (cid:12)(cid:12)(cid:12) w ( τ +1)
i (cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12) w ( τ )
i (cid:12)(cid:12)(cid:12) + ˜Θ( η ) yields (cid:12)(cid:12)(cid:12) W ( t +1)
[ i,j 0 ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) w ( t +1) 2 i (cid:12)(cid:12)(cid:12) = ˜Θ(1) .
Since we just proved ∀ i ∈ [ d ] : sign (cid:16) w ( t +1)
i (cid:17) = sign (cid:16) W ( t +1)
[ i, j 0 ] (cid:17) = sign (cid:16) w (0)2 i (cid:17) , we get that ( W 2 W 1 ) ( t +1) j= d (cid:88)
i = w ( t +1)
i
W ( t +1)
[ i, j 0 ] =
d (cid:88) i = (cid:12)(cid:12)(cid:12) w
[Table]
1) 2 i (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) W
[Table]
[ i, j 0 ] (cid:12)(cid:12)(cid:12) =
O (1) , which gives us that (cid:12)(cid:12)(cid:12) w ( t +1)
i (cid:12)(cid:12)(cid:12) =
˜ O (cid:16)
[Table]
d
[Table]
Recall that we have shown (cid:12)(cid:12)(cid:12) w
[Table]
(cid:12)(cid:12)(cid:12) ≥
[Table]
, then
[Table]
w
[Table]
(cid:12)(cid:12)(cid:12)
[Table]
Now we prove (cid:12)(cid:12)(cid:12) W ( t +1)
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:16) 1 √ d (cid:17) .
We have proved that T 1 ≤ τ ≤ t , ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) ∆ W (
[Table] [Table]
[Table]
, j ] (cid:12)(cid:12)(cid:12) =
[Table]
O
[Table]
η ) and (cid:12)(cid:12)(cid:12) w (
[Table]
1) 2 i (cid:12)(cid:12)(cid:12) −
[Table]
w
[Table]
=
[Table]
, then ∀
[Table] [Table]
∈
[Table]
d ] , (cid:12)(cid:12)(cid:12) W
[Table]
≤
[Table]
√ η + 1 d α − 1 / 2 (cid:1) .
Since we already proved that (cid:12) (cid:12) (cid:12) w ( t +1) 2 i (cid:12) (cid:12) (cid:12) =
˜ Θ (cid:16) 1 √ d (cid:17) , we get (cid:12) (cid:12) (cid:12) W
( t +1)(cid:12) (cid:12) (cid:12) =
˜ O (cid:16) 1 √ d (cid:17) .
Therefore by induction, for all t in the interval [ T 1 , ˜ T ) , we have ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) w ( t ) 2 i (cid:12)(cid:12)(cid:12) = ˜Θ (cid:16) 1 √ d (cid:17) , (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:16) 1 √ d (cid:17) .
From the proof we also get ∀ i ∈
[ d ] : (cid:12)(cid:12)(cid:12) w ( t +1) 2 i (cid:12)(cid:12)(cid:12) > (cid:12)(cid:12)(cid:12) w ( t ) 2 i (cid:12)(cid:12)(cid:12) , and that (cid:12)(cid:12)(cid:12) ∆ w ( t ) 2 i (cid:12)(cid:12)(cid:12) = ˜Θ( η ) , (cid:12)(cid:12)(cid:12) ∆ W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O ( η ) .
Now we verify that ˜ T
= ˜Θ (cid:16) 1 √ dη (cid:17) .
Combining ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) w ( ˜ T ) 2 i (cid:12)(cid:12)(cid:12) = ˜Θ (cid:16) 1 √ d (cid:17) and ∀ t ∈ [ T 1 , ˜ T ) , (cid:12)(cid:12)(cid:12) w ( t +1)
2 i (cid:12)(cid:12)(cid:12) − (cid:12)(cid:12)(cid:12) w ( t ) 2 i (cid:12)(cid:12)(cid:12) = ˜Θ( η ) , we immediately get that ˜ T − T 1 = ˜Θ (cid:16) 1 √ dη (cid:17) .
In Section D.2 we have shown that T 1 = Θ (cid:16) 1 √ dη (cid:17) , then we get ˜ T
= ˜Θ (cid:16) 1 √ dη (cid:17) .
D. P ROOF OF L EMMA
We prove this lemma by induction.
The base case ( t = T 1 ) can be veriﬁed by Lemma 14.
Now suppose for t in the interval [ T 1 , ˜ T ) , we have ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ Ω (cid:16) 1 √ d (cid:17) .
For t ∈ [ T 1 , ˜ T ) , by the proof of Lemma 26 (Section D.13), we know that for ∀ τ ≤ t , ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) w ( τ ) 2 i (cid:12)(cid:12)(cid:12) = ˜Θ (cid:16) 1 √ d (cid:17) , (cid:12)(cid:12)(cid:12) W ( τ )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:16) 1 √ d (cid:17) and that (cid:12)(cid:12)(cid:12) g ( τ )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) 1 √ d (cid:17) , (cid:12)(cid:12)(cid:12) g
( τ ) 2 i (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) √ d (cid:17) , and that ˜ T
= ˜Θ (cid:16) 1 √ dη (cid:17) .
Then we can pick H :
= 1 1 − β 1 log dηξ 2 and apply Lemma 16 and Corollary 2 to get that, w.h.p., for all t ∈
[ T 1 , ˜ T ) and ∀ i, j ∈ [ d ] , the update of W 1 can be written as
[Equation]
t
[Equation] [Equation]
[Equation]
ξ , where (cid:12)(cid:12)(cid:12) (cid:15) ( t ) 1 n [ i, j ] (cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12) (cid:15) ( t )
d
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O ( ηξ 2 ) .
By Lemma 23, we have that for 1 ≤ i, j ≤ d , g ( t )
[ i, j ] = w ( t )
i E ( t ) j = c ( t ) sign (cid:16) w (0)2 i (cid:17) E ( t )
j + R ( t ) g, 1 [ i, j ] ,
[Equation]
⇒
[Equation]
Using the fact that for a, b ∈ R , | a 2 − b 2 | a≤
[Equation]
|
[Equation]
2 , we get that (cid:16)
[Equation]
i
[Equation] [Equation]
=
[Equation] [Equation]
i
[Equation] [Equation]
[Equation]
[ i, j ] (cid:17)
[Equation]
(cid:16) c ( t ) E ( t )
[Equation]
(cid:17) 2 + R ( t ) gsqr , 1 [ i, j ] , where (cid:12)(cid:12)(cid:12) R ( t ) gsqr , 1 [ i,j ] (cid:12)(cid:12)(cid:12) (cid:16) c
( t ) E ( t ) j (cid:17) 2 =
[Equation]
O
[Equation]
1 / 2 (cid:1) .
That yields
[Equation] [Equation]
Since (cid:16) c ( t − τ ) E ( t − τ ) j (cid:17) 2 > 0 , in eq.
(38) we have that
[Equation]
However we cannot similarly prove that (cid:12)(cid:12)(cid:12)(cid:80)
Hτ =
β τ 1 R ( t − τ ) g,[ i, j ] (cid:12)(cid:12)(cid:12) (cid:28) (cid:12)(cid:12)(cid:12)(cid:80)
Hτ =
β τ 1 c ( t − τ ) E
( t − τ ) j (cid:12)(cid:12)(cid:12) in eq.
(37) because c ( t − τ ) E ( t − τ )
j may not have the same sign for τ = 0 , 1 , ..., H .
To deal with eq.(37), we need to consider the two cases where (cid:12)(cid:12)(cid:12)(cid:80)
Hτ =
β τ 1 R ( t − τ ) g,[ i, j ] (cid:12)(cid:12)(cid:12) (cid:28) (cid:12)(cid:12)(cid:12)(cid:80)
Hτ =
β τ 1 c ( t − τ ) E
( t − τ ) j (cid:12)(cid:12)(cid:12) or (cid:12)(cid:12)(cid:12)(cid:80)
[Table]
g,[ i,
[Table]
j (cid:12)(cid:12)(cid:12) .
[Table]
[Equation]
where
[Equation]
Note that from eq.
(39) we have
[Equation]
Combining with
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
ξ 2 , we can apply Lemma 33 to get that
W
[Equation]
[Table]
, where (cid:12)(cid:12)(cid:12) e ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:16)
η 14 + d α 2 −(cid:17) .
Since (cid:12)(cid:12)(cid:12) W ( t +1)
[ i, j ] − W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O ( η ) , we get that (cid:12) (cid:12) (cid:12) v ( t ) j
(cid:12) (cid:12) (cid:12) =
˜ O ( η ) .
Case 2.
[Equation]
[ i, j ] +
[Equation]
E
[Equation]
(cid:12)(cid:12)(cid:12)
where δ = (cid:16) η 14 + d
[Equation]
−(cid:17) .
Since (cid:12)(cid:12)(cid:12) R ( t ) g, 1 [ i,j ] (cid:12)(cid:12)(cid:12) c ( t ) (cid:12)(cid:12)(cid:12) E ( t ) j (cid:12)(cid:12)(cid:12) =
˜ O (cid:0) √ η + 1 d
α − 1 / 2 (cid:1) , we have that (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (1 − β 1 )
H (cid:88) τ = β τ 1 R ( t
[Equation]
[Equation]
,[ i, j ] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation] [Equation]
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)
[Equation] [Equation]
H
[Equation]
= β τ
[Equation]
E ( t
[Equation]
) j (cid:17)
[Equation]
(cid:19) (cid:118)(cid:117)(cid:117)(cid:116) (
[Equation] [Equation]
(cid:88)
[Equation]
=
[Equation]
(cid:16)
[Equation] [Equation]
] (cid:17)
[Equation]
, where ( i ) uses Cauchy-Schwarz inequality and β 2 = β 21 , ( ii ) uses eq.
(38) and (39).
Combining with (cid:12) (cid:12)(cid:12) (cid:15) ( t ) 1 n [ i, j ] (cid:12) (cid:12)(cid:12) ≤ ˜ O ( ηξ 2 ) ≤ ˜ O (cid:0) √ η + 1 d α − 1 / 2 (cid:1) (cid:18) ξ − (cid:114)(cid:12) (cid:12)(cid:12) (cid:15) ( t ) 1 d [ i, j ] (cid:12) (cid:12)(cid:12) (cid:19) gives us (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (1 − β 1 )
H (cid:88) τ = β τ 1 c
( t − τ ) E ( t − τ ) j (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) < (cid:12)(cid:12)(cid:12) (1 − β 1 ) (cid:80)
Hτ =
β τ 1 R ( t − τ ) g,[ i, j ] + (cid:15) ( t ) 1 n [ i, j ] (cid:12)(cid:12)(cid:12) η 14 + d α 2 −≤ ˜ O (cid:0) √ η + 1 d α − 1 / 2 (cid:1)
η 14 + d α 2 −  (cid:118)(cid:117)(cid:117)(cid:116)
H (cid:88) τ
= β τ 2 (cid:16) g ( t − τ )
[ i, j ] (cid:17) 2 − (cid:114)(cid:12)(cid:12)(cid:12) (cid:15) ( t )
d [ i, j ] (cid:12)(cid:12)(cid:12) + ξ   ≤ ˜ O (cid:18) η 1 4 + d α 2 −(cid:19)   (cid:118)(cid:117) (cid:117) (cid:116)
H (cid:88) τ = β τ 2 (cid:16) g
( t − τ )
[ i, j ] (cid:17) 2 + (cid:15) ( t )
d [ i, j ] + ξ   , which implies (cid:12)(cid:12)(cid:12) W ( t +1)
[ i, j ] − W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
Consider certain i.j ∈ [ d ] and the period from T 1 to t .
Denote T as the set of time points when Case 1 is satisﬁed.
By Lemma 26, we know that η ( t − T 1 ) =
˜ O (cid:16)
[Equation]
d (cid:17) , which gives us (cid:88)
[Equation]
By the ﬁrst phase analysis, we have that
[Equation]
where V ( T 1 ) j
= O (cid:16) 1 √ d (cid:17) ,
[Equation]
[ i, j
[Equation]
ηd +
[Equation]
d α (cid:1) .
Combining with the analysis of Case 1, we have that
[Equation]
where | R T [ i, j ] | ≤ O (cid:16) η 14 +
[Equation]
(cid:16)(cid:12) (cid:12) (cid:12) V ( T 1 )
j
[Equation]
(cid:12) (cid:12) + (cid:80)
[Equation]
(cid:12)
[Equation]
(cid:12) (cid:12) (cid:17) .
Since for τ ∈ T , (cid:12)(cid:12)(cid:12) v ( τ ) j (cid:12)(cid:12)(cid:12) =
˜ O ( η ) , V ( T 1 )
j
= O (cid:16) 1 √ d (cid:17) and η ( t − T 1 ) =
˜ O (cid:16) 1 √ d (cid:17) , we can bound | R T [ i, j ] | by
[Equation]
Combining the above results together yields
[Equation]
+ R ( t )
[ i, j ] , where (cid:12)(cid:12)(cid:12) R ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:32)
ηd
12 + d α 2 +(cid:33) .
By the inductive hypothesis (cid:12)(cid:12)(cid:12) W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) = ˜Ω (cid:16) 1 √ d (cid:17) , we get that (cid:12)(cid:12)(cid:12) V ( t ) j
(cid:12)(cid:12)(cid:12) =
˜Ω (cid:16) 1 √ d (cid:17) , which gives us (cid:12)(cid:12)(cid:12) R ( t ) 1 [ i,j ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) V ( t ) j (cid:12)(cid:12)(cid:12) ≤ ˜ O (cid:16) η 14 + d α 2 −(cid:17) .
Therefore, we have that for any j ∈ [ d ] and any i 1 , i 2 ∈ [ d ] , (cid:12)(cid:12)(cid:12) W ( t )
[ i 1 , j ] (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) W ( t )
[ i 2 , j ] (cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12) sign (cid:16) w (0)2 i 1 (cid:17) V ( t ) j (cid:16) 1 ± ˜ O (cid:16)
η 14 + d α 2 − 14 (cid:17)(cid:17)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) sign (cid:16) w (0)2 i 2 (cid:17) V ( t ) j (cid:16) 1 ± ˜ O (cid:16)
η 14 + d α 2 − 14 (cid:17)(cid:17)(cid:12)(cid:12)(cid:12) =
1 ± ˜ O (cid:18)
η 14 + d α 2 −(cid:19) .
By Lemma 23, we know that (cid:12)(cid:12)(cid:12) w ( t ) 2 i (cid:12)(cid:12)(cid:12) with different i are also roughly equal, i.e. (cid:12)(cid:12)(cid:12) w ( t ) 2 i 1 (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) w ( t ) 2 i 2 (cid:12)(cid:12)(cid:12) = 1 ±
˜ O (cid:0)
√ η + 1 d
α − 1 / 2 (cid:1) .
Then we have for any j ∈ [ d ] , ( W 2 W 1 ) ( t ) j = d (cid:88) i = w ( t )
i W ( t )
[ i, j ] =
d (cid:88) i = (cid:12)(cid:12)(cid:12) w ( t ) 2 i (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
Θ (cid:16) d (cid:12)(cid:12)(cid:12) w ( t ) 2 k (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) W ( t ) 1 [ k, j ] (cid:12)(cid:12)(cid:12)(cid:17) = ˜Θ (cid:16) √ d (cid:12)(cid:12)(cid:12) W ( t ) 1 [ k, j ] (cid:12)(cid:12)(cid:12)(cid:17) .
where k can be any index in { 1 , 2 , ..., d } and the last equality uses ∀
i ∈
[ d ] : (cid:12)(cid:12)(cid:12) w ( t ) 2 i (cid:12)(cid:12)(cid:12) =
˜ Θ (cid:16) 1 √ d (cid:17) .
Now we analyze the lower bound of (cid:12)(cid:12)(cid:12) W ( t +1)
[ k, j ] (cid:12)(cid:12)(cid:12) .
Although it may decrease during some period, we observe that once (cid:12)(cid:12)(cid:12) W ( t ) 1 [ k, j ] (cid:12)(cid:12)(cid:12) decreases to some value of order ˜Θ (cid:16) 1 √ d (cid:17) such that ( W 2 W 1 ) ( t )
j <
A j − √ ηd , i.e. E ( t )
j < −√ ηd , we can apply the technique in Section D.13 when analyzing
W ( t )
[ i, j 0 ] to get that (cid:12)(cid:12)(cid:12) W ( t ) 1 [ k, j ] (cid:12)(cid:12)(cid:12) will increase in the next step.
This mechanism ensures a ˜Ω (cid:16) 1 √ d (cid:17) lower bound of (cid:12)(cid:12)(cid:12) W ( t +1)
[ k, j ] (cid:12)(cid:12)(cid:12) .
Since k, j are arbitrary, we have proved that at time t + 1 , ∀ i, j ∈ [ d ] : (cid:12)(cid:12)(cid:12) W ( t +1)
[Equation]
Therefore by induction, we conclude that when T 1 ≤ t < ˜ T , for ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜Ω (cid:16) 1 √ d (cid:17) .
The remaining part of this lemma has also been proved by the analysis above.
[Equation] [Equation]
Lemma 26 tells us that for any i ∈ [ d ] , (cid:12)(cid:12)(cid:12) w ( t )
i (cid:12)(cid:12)(cid:12) keeps increasing when t < ˜ T .
However, the behavior of W ( t ) 1 [ i, j ] is more complicated.
The following lemma tells us that (cid:12) (cid:12) (cid:12) W ( t )
[ i, j ] (cid:12) (cid:12) (cid:12) will increase until
T f,j .
After that (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) and E ( t )
j may zigzag, but E ( t )
j will not ﬂuctuate dramatically and will be trapped in a small interval around zero.
Lemma 27.
Under Assumption 1, 2 and 3, suppose σ ≤ η 3 / 2 ξ 2 d 13 / 4 .
Pick η ≤ O (cid:0) 1 d 3 α (cid:1) , ξ ≤ (cid:112) η d 3 α − 1 , and β 2 = β 21 .
Consider certain coordinate j .
For T 1 ≤ t < min (cid:110) ˜ T, T f,j (cid:111) , we have ∀ i ∈
[ d ] : (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) keeps increasing.
If T f,j < ˜ T , then for T f,j ≤ t < ˜ T , we will have − ˜ O (cid:0) √ ηd (cid:1) ≤ E ( t ) j ≤ ˜ O
(cid:0) √ ηd (cid:1) .
Now we start proving Lemma 25.
At time ˜ T , denote S :=
(cid:110)
j : T f,j < ˜ T (cid:111) , i.e. the set of coordinates whose E j have passed its “ﬂip time”.
By Lemma 27, we know that ∀ j ∈ S, (cid:12)(cid:12)(cid:12) E ( ˜ T ) j (cid:12)(cid:12)(cid:12) ≤ ˜ O
(cid:0) √ ηd (cid:1) .
If
S c = φ , which means ∀ j ∈
[ d ] : (cid:12)(cid:12)(cid:12) E ( ˜ T ) j (cid:12)(cid:12)(cid:12) ≤ ˜ O
(cid:0) √ ηd (cid:1) , then our lemma will immediately follow.
If
S c (cid:54) =
φ , we have ˜ T
= min { T g , T f } =
T g and that ∀ j ∈ S c : E ( ˜ T )
j < 0 .
By the deﬁnition of T g , we know that ∃ i 0 ∈
[ d ] : (cid:12)(cid:12)(cid:12) g ( ˜ T ) 2 i 0 (cid:12)(cid:12)(cid:12) ≤ O (cid:0) d √ η (cid:1) .
Then (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:12) d (cid:88) j ∈ S c
E
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:12) = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:12) d (cid:88)
[Equation]
=
[Equation]
(cid:12) ≤ (cid:12)(cid:12)(cid:12) g
[Equation]
[Equation]
0 (cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:88)
[Equation]
d
[Equation]
O
[Equation]
d √ η ) .
By Lemma 24, we know that when T 1 ≤ t < ˜ T , for ∀
i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) =
˜Ω (cid:16) 1 √ d (cid:17) .
Since the update per step (cid:12)(cid:12)(cid:12) ∆ W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O
( η ) , we know that sign (cid:16) W ( t )
[ i, j ] (cid:17) remains unchanged during this period and sign (cid:16) W ( t )
[ i, j ] (cid:17)
= sign (cid:16) W ( T 1 )
[ i, j ] (cid:17)
= sign (cid:16) w (0)2 i (cid:17) independent of j .
Combining with ∀ j ∈ S c : E ( ˜ T )
j < 0 gives us that E ( ˜ T ) j W ( ˜ T ) 1 [ i 0 , j ] for different j have the same sign.
Therefore for any j 0 ∈ S c , ˜ O
[Equation]
[Table]
Note that the above inequality holds for any j 0 ∈ S c , which means ∀ j ∈ S c : (cid:12)(cid:12)(cid:12) E ( ˜ T ) j (cid:12)(cid:12)(cid:12) ≤ ˜ O
(cid:0) d √ ηd (cid:1) .
Combining with the fact that ∀ j ∈ S : (cid:12)(cid:12)(cid:12) E ( ˜ T ) j (cid:12)(cid:12)(cid:12) ≤ ˜ O
(cid:0) d √ ηd (cid:1) completes the proof.
D. P ROOF OF L EMMA
Consider certain j ∈
[ d ] , when t < min (cid:110) ˜ T, T f,j (cid:111) , we have that E ( t )
j < −√ ηd .
Therefore we can use the same argument as in Section D.13 to prove that (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) keeps increasing, and sign (cid:16) W ( t )
[ i, j ] (cid:17)
= sign (cid:16) w (0)2 i (cid:17) for all i ∈
[ d ] .
At time the “ﬂip time” t =
T f,j , by deﬁnition, E ( t ) j ≥ −√ ηd .
After that E ( t ) j may oscillate.
Now we prove that once E ( t ) j ≥ √ ηd (or E ( t ) j ≤ −√ ηd ), after a short period E ( t )
j will decrease (or increase) until E ( t ) j ≤ ˜ O
(cid:0) √ ηd (cid:1) (or E ( t )
j ≥ − ˜ O (cid:0) √ ηd (cid:1) ).
Moreover, during this period, E ( t ) j won’t change too much.
We ﬁrst recall that when T 1 ≤ t < ˜ T , Lemma 26 gives us for all i ∈ [ d ] , (cid:12)(cid:12)(cid:12) w ( t )
i (cid:12)(cid:12)(cid:12) = ˜Θ (cid:16) 1 √ d (cid:17) and (cid:12)(cid:12)(cid:12) W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) =
˜ O (cid:16) 1 √ d (cid:17) .
Then eq.(35)
we obtained in the ﬁrst phase analysis still holds, which tells us that the change of E ( t ) j per step satisﬁes (cid:12)(cid:12)(cid:12) E ( t +1)
j
[Equation]
for all T
[Equation]
≤ t < ˜ T .
We divide the analysis into two cases, based on whether these E ( t ) j ≥ √ ηd or E ( t ) j ≤ −√ ηd .
By
Lemma 24, we know that when T 1 ≤ t < ˜ T , ∀ i ∈
[ d ] , (cid:12)(cid:12)(cid:12) W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) = ˜Ω (cid:16) 1 √ d (cid:17) .
Since the update per step (cid:12)(cid:12)(cid:12) ∆ W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) ≤ ˜ O
( η ) , we know that sign (cid:16) W ( t )
[ i, j ] (cid:17) remains unchanged during this period and sign (cid:16) W ( t )
[ i, j ] (cid:17)
= sign (cid:16) W ( T 1 )
[ i, j ] (cid:17)
= sign (cid:16) w (0)2 i (cid:17) .
By the analysis of w ( t ) 2 i in Lemma 23, we have for all i ∈
[ d ] , w ( t +1) 2 i = w ( t ) 2 i + sign (cid:16) w (0)2 i (cid:17) ∆ ( t ) 2 i , where ∆ ( t )
i
=
η (cid:16) 1 ± ˜ O (cid:0) √ η (cid:1)(cid:17) .
Case 1.
Consider some time point t such that E ( t ) j ≤ −√ ηd .
Note that for all i ∈ [ d ] , (cid:12)(cid:12)(cid:12)
[Equation]
t
[Equation]
1 [ i, j ]
[Equation]
=
[Equation]
t ) 2 i E ( t ) j (cid:12)(cid:12)(cid:12) = ˜Ω (cid:0) √ η (cid:1) and that sign (cid:16) g ( t )
[ i, j ] (cid:17) =
− sign (cid:16) w ( t ) 2 i (cid:17) = − sign (cid:16) w (0)2 i (cid:17) .
By Lemma 23, for all i ∈ [ d ] we have W ( t +1)
[ i, j ] = W ( t )
[ i, j ] + sign (cid:16) w (0)
i (cid:17) ∆ ( t )
[ i, j ] with ∆ ( t )
[ i, j ] =
η (cid:16) 1 ± ˜ O (cid:0) √ η (cid:1)(cid:17) .
That gives us
[Equation] [Equation]
∆ ( t ) 1 [ i, j ] (cid:17) − A j = d (cid:88) i = (cid:16) w ( t ) 2 i W ( t )
[ i, j ] + sign (cid:16) w (0) 2 i (cid:17) (cid:16) w ( t ) 2 i ∆ ( t ) 1 [ i, j ] + ∆ ( t ) 2 i W ( t )
[ i, j ] (cid:17) + ∆ ( t ) 2 i ∆ ( t )
[ i, j ] (cid:17) − A j ( i ) =
E ( t ) j + d (cid:88) i = (cid:16)(cid:12)(cid:12)(cid:12) w ( t )
i (cid:12)(cid:12)(cid:12) ∆ ( t ) 1 [ i, j ] + ∆ ( t ) 2 i (cid:12)(cid:12)(cid:12) W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) + ∆ ( t ) 2 i ∆ ( t ) 1 [ i, j ] (cid:17) , ⇒ E ( t +1)
j > E ( t ) j , where ( i ) is because sign (cid:16) w ( t ) 2 i (cid:17) = sign (cid:16) W ( t )
[ i, j ] (cid:17)
= sign (cid:16) w (0)2 i (cid:17) .
Therefore we have proved that E ( t ) j will increase in the next step.
After that for τ ≥ t + 1 , as long as E ( τ ) j ≤ −√ ηd , the above analysis will hold and E ( τ ) j will keep increasing until E ( τ ) j > −√ ηd
or we reach ˜ T .
Case 2.
Consider some time point t such that E ( t ) j ≥ √ ηd .
We will prove that E ( t ) j will decrease after a short period, and during this period, the change of it is at most ˜ O (cid:0) √ ηd (cid:1) .
By similar arguments as in Case 1, we can get that W ( t +1)
[ i, j ] = W ( t )
[ i, j ] − sign (cid:16) w (0)
[Equation]
(cid:17) ∆ (
[Equation] [Equation]
[Equation]
, where ∆ ( t ) 1 [ i, j ] =
η (cid:16) 1 ± ˜ O (cid:0) √ η (cid:1)(cid:17) , Then
E ( t +1)
j = d (cid:88)
i = w ( t +1)
i
W ( t +1)
[ i, j ] − A j =
[Equation]
(cid:88) i = (cid:16) w ( t )
i + sign (cid:16) w (0)2 i (cid:17) ∆ ( t ) 2 i (cid:17) (cid:16) W ( t )
[ i, j ] − sign (cid:16) w (0)2 i (cid:17) ∆ ( t )
[ i, j ] (cid:17) − A j = d
[Equation]
i = (cid:16) w
[Equation]
W (
[Equation]
1 [ i, j ] − sign (cid:16) w (0)2 i (cid:17)
[Table]
w
[Table]
2 i ∆ ( t )
[ i, j ] − ∆ (
[Table]
) 2 i W (
[Table]
1 [ i, j ]
[Table]
− ∆
[Table] [Table]
∆
[Table]
) 1 [ i, j ] (cid:17) − A j ( i ) =
E ( t ) j − d (cid:88)
i = (cid:16)(cid:12)(cid:12)(cid:12) w ( t )
i (cid:12)(cid:12)(cid:12) ∆ ( t ) 1 [ i, j ] − ∆ ( t )
i (cid:12)(cid:12)(cid:12) W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) + ∆ ( t ) 2 i ∆ ( t ) 1 [ i, j ] (cid:17) , where ( i ) is because sign (cid:16) w ( t )
i (cid:17) = sign (cid:16) W ( t )
[ i, j ] (cid:17)
= sign (cid:16) w (0)2 i (cid:17) .
E ( t +1)
j may not be smaller than E ( t ) j , but we will show that after at most t s steps for some t s , we will have E ( t + t s +1)
j < E ( t + t s ) j .
To see this, ﬁrst note that by the bounds of ∆ ( t ) 1 [ i, j ] and ∆ ( t )
i , we get ∆ ( t )
[ i, j ] ≥ ∆ ( t ) 2 i − η ˜ O (cid:0) √ η (cid:1) .
Since (cid:12)(cid:12)(cid:12) w ( t )
i (cid:12)(cid:12)(cid:12) increases by ˜Θ( η ) per step, and (cid:12)(cid:12)(cid:12) W ( t )
[ i, j ] (cid:12)(cid:12)(cid:12) keeps decreasing, then we have either i) after t s steps for some t s , ∀ i ∈
[ d ] : (cid:12)(cid:12)(cid:12) w ( t + t s ) 2 i (cid:12)(cid:12)(cid:12) ≥ (cid:12)(cid:12)(cid:12) W ( t + t s )
[ i, j ] (cid:12)(cid:12)(cid:12) + √ η or ii) we reach ˜ T .
For i), if E ( t + t s )
j < √ ηd , then it’s already what we want.
Otherwise we will have ∆ ( t + t s )
[ i, j ] =
η (cid:16) 1 ± ˜ O (cid:0) √ η (cid:1)(cid:17) .
Hence
E ( t + t s )
j − E ( t + t s +1)
j = d (cid:88)
i = (cid:16)(cid:12)(cid:12)(cid:12) w ( t + t s )
i (cid:12)(cid:12)(cid:12) ∆ ( t + t s )
[ i, j ] − ∆ ( t + t s )
i (cid:12)(cid:12)(cid:12) W ( t + t s )
[ i, j ] (cid:12)(cid:12)(cid:12) + ∆ ( t + t s ) 2 i ∆ ( t + t s )
[ i, j ] (cid:17) ≥ d (cid:88)
i = (cid:16)(cid:12)(cid:12)(cid:12) W ( t + t s )
[ i, j ] (cid:12)(cid:12)(cid:12) (cid:16) ∆ ( t + t s )
[ i, j ] − ∆ ( t + t s )
i (cid:17) + √ η ∆ ( t + t s )
[ i, j ] + ∆ ( t + t s ) 2 i ∆ (
[Equation]
>
[Equation]
where the last inequality uses ∀ i, j ∈
[ d ] : (cid:12)(cid:12) (cid:12) W ( t + t s )
[ i, j ] (cid:12)(cid:12) (cid:12) =
˜ O (cid:16) 1 √ d (cid:17) .
Therefore E ( t + t s +1)
j <
E ( t + t s ) j .
After that for τ ≥ t + t s + 1 , as long as E ( τ ) j ≥ √ ηd , the above analysis will hold and E ( τ ) j will keep decreasing until E ( τ ) j < √ ηd or we reach ˜ T .
Now we prove that during these t s steps, the change of E j is ˜ O (cid:0) √ ηd (cid:1) .
Since at each step the difference | w 2 i |−| W
[ i, j ] | will be enlarged by ˜ Ω( η ) , then we know that t s = √ η/ ˜ Ω( η ) =
˜ O (cid:16) 1 √ η (cid:17) .
Combining with the fact that for all T 1 ≤ τ ≤ ˜ T , (cid:12)(cid:12)(cid:12) E ( τ +1)
j − E ( τ ) j (cid:12)(cid:12)(cid:12) =
˜ O (cid:16) η √ d (cid:17) gives us
[Equation]
For ii), we reach ˜ T before ∀ i ∈
[ d ] : (cid:12)(cid:12)(cid:12) w ( t + t s ) 2 i (cid:12)(cid:12)(cid:12) ≥ (cid:12)(cid:12)(cid:12) W ( t + t s )
[ i, j ] (cid:12)(cid:12)(cid:12) + √ η .
Then we have ˜ T − t ≤ √ η/ ˜Ω( η ) =
˜ O (cid:16) 1 √ η (cid:17) , which yields E ( ˜ T )
j
[Equation]
Combining the above two cases, we ﬁnd that if for some t , E ( t ) j ≥ √ ηd , then after at most t s steps
E j will decrease and keeps decreasing until E j < √ ηd or we reach ˜ T .
During these steps, E j can increase at most ˜ O (cid:0) √ ηd (cid:1) .
If for some t , E ( t ) j ≤ −√ ηd , then after one step it will increase and keeps increasing until E j > √ ηd or we reach ˜ T .
That means once for some coordinate j , E j overshoots, it will zigzag in a small region around zero, which is (cid:104) − ˜ O
(cid:0) √ ηd (cid:1) ,
˜ O (cid:0)
√ ηd (cid:1)(cid:105)
[Equation]
E H ESSIAN TENDS TO BECOME MORE AND MORE DIAGONAL DURING TRAINING
In this section, we empirically demonstrate that the trend of loss Hessian in practice is to become more and more diagonal during training.
We also give a rigorous theoretical analysis on a two-layer network under Assumption 1 and 2.
E. E MPIRICAL R ESULTS
Let’s ﬁrst deﬁne the diagonal domination of the i -th coordinate at time t .
[Equation]
To measure the diagonal domination of the whole Hessian, we need to consider the distribution of r OPTdiag ,i ( t ) for different i .
Figure 14 shows the mean and median of r SGDMdiag ,i ( t ) and r Adamdiag ,i ( t ) on the sentence classiﬁcation task (see Section 4.1).
Here we chose 4 layers (Layer #6, 12, 17 and 22) and computed the Hessians across these 4 layers.
Since the number of parameters is very large, we did the computation by random sampling.
As we can see, for both r SGDMdiag ,i ( t ) and r Adamdiag ,i ( t ) , the trend of their mean or median is to decrease over time, although there might be some oscillation.
[Figure]
T HEORETICAL A NALYSIS
To simplify the theoretical analysis, we consider the mean of r OPTdiag ,i ( t ) over all coordinate and deﬁne
[Equation]
We consider a 2-layer network under Assumption 1 and 2, and have two goals in our proof:
To show that R OPTdiag ( t ) after training is smaller than that before training ( t = 0 ).
Note that in our setting (see in Assumption 1), the Hessian is a ( d 2 + d ) ×
( d 2 + d ) matrix.
For a completely “uniform” matrix with the same size, we have that R OPTdiag ( t ) =
Θ (cid:0) √
d 2 + d (cid:1) =
Θ( d ) .
Hence our second goal is to show that the R OPTdiag ( t ) after training is on lower order than Θ( d ) .
Theorem 2.
Consider the ratio R OPTdiag ( t ) deﬁned in eq.
Under Assumption 1 and 2, we have that before training ( t = 0 ), with high probability,
[Equation]
For SGD+M deﬁned in eq.
For any p > 0 , by picking the same hyperparameters as in Theorem 1, for T SGD , 1 , T SGD , 2 mentioned in Theorem 1, we have with constant probability, for any t ∈
[ T SGD , 1 , T SGD , 2 ] ,
[Equation]
42) where the trend of q ( t ) is to decrease over time and q ( T SGD
[Equation]
2 ) ≤
[Equation]
For Adam deﬁned in eq.
For any p > 0 , by picking the same hyperparameters as in Theorem 1, for
T Adam , 1 , T Adam , 2 mentioned in Theorem 1, we have with high probability, for any t ∈
[ T Adam , 1 , T Adam , 2 ] ,
[Equation]
(43) where the trend of r ( t ) is to decrease over time and r ( T Adam , 2 ) ≤ ˜ O (cid:16) 1 d p − 12 (cid:17) =
o (cid:16) √ d (cid:17) .
E. P ROOF OF T HEOREM
Lemma 4.3 of (Kawaguchi, 2016) gives us the following forms of Hessian.
For any k ∈ { 1 , 2 , ..., H + 1 } , we know that ∇ vec ( W k ) ( ∇ vec ( W k ) L ( W )) equals (( W H
+1 . . .
W k +1 ) T ( W H
+1 . . .
W k +1 )
⊗ ( W k − 1 . . .
W 1 )
( W k − 1 . . .
W 1 ) T , and for k ∈ { 2 , 3 , ..., H + 1 } , ∇ vec ( W k )
( ∇ vec ( W 1 ) L ( W )) =(
C T ( W H +1 . . .
W k +1 )
⊗ ( W k − 1 . . .
W 1 ) T )
+[( W k − 1 . . .
W 2 ) T ⊗ I ][ I d k − 1 ⊗ ( r ( W H +1 . . .
W k +1 )) .
, 1 · · · I d k − 1 ⊗ ( r ( W H +1 . . .
W k +1 )) .,d
k ] , where r =
( W H +1 . . .
W 1 − A ) T , C =
W H +1 W H · · · W 2 .
For the 2-layer linear network, write the Hessian as
[Equation] [Equation]
, then we have that
[Equation]
Intuitively, before training the elements of W 1 and W 2 are very close to zero, and W 2 W 1 − A ≈ − A .
Since the elements of A are Θ(1) , we know that the magnitudes of elements of H 21 are much bigger than those of H 11 and H 22 .
After training, for both SGD+M and Adam, W 2 W 1 − A ≈ 0 .
Then H
≈ ( W 2 ) T ⊗ ( W 1 ) T and the magnitudes of its elements are no longer much larger than those of H 11 and H 22 .
From the formula of H 11 , we know that all the diagonal entries are nonzero, and among the d 4 − d 2 off-diagonal entries, there are only d 3 − d 2 nonzero entries, which helps us to bound R OPTdiag ( t ) .
E.3. P ROOF OF EQ . (41)
Let’s ﬁrst analyze the weights and Hessian before training ( t = 0 ).
For ease of notation, we omit the superscript ( t ) .
For the i -th row where 1 ≤ i ≤ d , i.e. the i -th row of the submatrix [ H
H T 21 ] , we have (cid:88)
[Equation]
On the other hand, for the diagonal elements, we have w.h.p.
[Equation]
Then we have that for 1 ≤ i ≤ d ,
[Equation]
For the ( id + k ) -th row where 1 ≤ i ≤ d, 1 ≤ k ≤ d , i.e. the (( i − 1) d + k ) -th row of the submatrix [ H
H 11 ] , we have (cid:88)
[Equation]
H
[Equation]
[ i
[Equation]
j ]
[Equation]
(cid:88)
[Equation]
Θ(1) .
On the other hand, for the diagonal elements, we have w.h.p.
[Equation]
Then we have that for 1 ≤ i ≤ d, 1 ≤ k ≤ d ,
[Equation]
Taking the average, we obtain that before training, i.e. when t = 0 ,
[Equation]
E.3. P ROOF OF EQ . (42)
The proof is based on the lemma below.
Lemma 28.
Suppose the weight matrices have the following structure:
[Equation]
where ∀ 1 ≤ i, j ≤ d :
[Equation]
Then
[Equation]
j
[Equation]
=
i
[Equation]
and for 1 ≤ i ≤ d, 1 ≤ k ≤ d ,
[Equation]
Now we are ready to prove eq.
By the analyses in Section C.1, we know that for t ∈ [ T SGD , 1 , T SGD , 2 ] , the weights obtained by GD with momentum satisfy
[Equation]
where T SGD , 1 = T 1 and ∀
[Equation]
≤ i,
[Equation]
≤ d :
[Equation]
Here (cid:15) 0 is deﬁned in Deﬁnition 2.
Since u ( T 1 ) doesn’t depend on time t in the period ( T SGD , 1 , T SGD , 2 ] , we write u ( T 1 ) as u for ease of notation.
Hence by Lemma 28, when t ∈ [ T SGD , 1 , T SGD , 2 ] , we have for 1 ≤ i ≤ d , (cid:113)(cid:80) j (cid:54) =
i (cid:0) H
[Equation] [Equation]
[Equation] [Equation]
[Equation]
and for 1 ≤ i ≤ d, 1 ≤ k ≤ d , (cid:113)(cid:80) j (cid:54) = id + k (cid:0) H
[Equation]
t )
[ i
[Equation] [Equation]
(cid:115)(cid:80)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
( t
[Equation]
(cid:12)(cid:12)(cid:12) (cid:16)
[Equation]
( (cid:15) 0 ) (cid:17)
[Equation]
(cid:0)
[Equation]
t
[Equation]
i
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]


[Equation]
By Lemma 3, we have u = X + Y where X i , i ∈
[ d ] are i.i.d Gaussian random variables and w.h.p.,
[Equation]
[Equation]
(46) which yields that ∀ i ∈ [ d ] :
[Equation]
By the proof in Section C.8, we know that for t ∈ [ T SGD , 1 , T SGD , 2 ] , ∀ i ∈
[ d ] : v ( t ) i , c ( t ) are positive.
The induction in Section C.9 further gives us that for t ∈ [ T SGD , 1 , T SGD , 2 ] , w.h.p.
∀ k ∈ [ d ] : v ( t )
k c
( t ) =
Θ (cid:16) 1 √ d (cid:17) , which yields c
[Equation]
(cid:107) v ( t ) (cid:107) 2 = Θ(1) .
Combining with eq.
(47), we obtain
[Equation]
  1 + (cid:12)(cid:12)(cid:12) v ( t ) k (cid:12)(cid:12)(cid:12) (cid:12)(cid:12) c ( t
[Equation]
(cid:12)(cid:12) 
 (cid:115)(cid:80) dj
[Equation] [Equation]

 .
By the proof in Section C.8, we know that for t ∈ [ T SGD , 1 , T SGD , 2 ] , ∀ i ∈
[ d ] : v ( t ) i , c ( t ) are positive and monotonically increasing.
On the other hand, the proof in Section C.2 and C.9 tells us that w.h.p.
[Equation]
resp
[Equation]
t )
[Equation]
) decreases from Θ( √ d ) (resp. Θ(1) )
when t = T SGD , 1 to O ( √ (cid:15) 0 d ) (resp.
O ( √ (cid:15) 0 ) )
when t = T SGD , 2 .
Therefore, the trend of (cid:107) E ( t ) (cid:107) 2 ui (cid:107) v ( t ) (cid:107) 2 2 and (cid:12)(cid:12)(cid:12) E ( t ) k (cid:12)(cid:12)(cid:12) ( c ( t ) ) 2 u 2 i is to decrease over time, and when t = T SGD , 2 , we have w.h.p.
[Equation]
Moreover, when t = T SGD , 2 , the inequality in eq.
(26) becomes equality, i.e. c 2 (cid:107) u (cid:107) 22 = Θ (cid:16) √ d (cid:17) and ∀ j ∈ [ d ] :
[Equation]
u (cid:107) 22 v 2 j = Θ
[Equation]
d (cid:17) .
Using u = X + Y and eq.
[Equation]
we have
[Equation]
which together with the second inequality in eq.
(47) yields
[Equation]
Combining with eq.
(49), we get that
[Equation]
Substituting eq.
(48) and (50) into eq.
(44) and (45) gives us
[Equation]
where the trend of q ( t ) 1 i is to decrease over time and q ( T SGD , 2 )
i
[Equation]
We also have
[Equation]
where the trend of q ( t ) 2 i is to decrease over time and q ( T SGD , 2 )
i
[Equation]
Hence
[Equation]
d j =1 X 2 j | X i | 
 + d 2 + d d (cid:88) i =
q ( t ) 1 i + d d 2 + d d (cid:88) i =
q ( t )
i :=
O   1 d d (cid:88)
i =
(cid:113)(cid:80) dj
=1 X 2 j | X i | 
 + q ( t ) , where the trend of q ( t ) is to decrease over time and q
[Equation]
Denote σ 2 as the variance of X i for i ∈ [ d ] .
By concentration of chi-squared distribution, we know that with probability at least 1 − δ for δ > 0 , d (cid:88) i =
X
[Equation]
≤ σ
[Equation]
O
[Equation]
By Lemma 35 in Appendix G, we know that with constant probability 1 d (cid:80) di =1 1 | X i | =
O (cid:0) 1 σ log d (cid:1) .
Then with constant probability, 1 d (cid:80) di =1 1 X 2 i ≤ 1 d (cid:16)(cid:80) di =1 1 | X i | (cid:17) 2 =
O (cid:0) dσ 2 log 2 d (cid:1) .
Hence d d (cid:88) i
= (cid:113) (cid:80)
[Equation]
Therefore with constant probability,
[Equation]
where the trend of q ( t ) is to decrease over time and q ( T SGD , 2 ) ≤ ˜ O (cid:0) d √ (cid:15)
d (cid:1) .
For any p > 0 , by picking the same hyperparameters as in Theorem 1, we have (cid:15)
d ≤ ˜ O (cid:0) 1 d p (cid:1) and hence q ( T SGD , 2 ) ≤ ˜ O
(cid:0)
[Equation]
E.3. P ROOF OF EQ . (43)
By the analyses in Section D.1, we know that for t ∈ [ T Adam , 1 , T Adam , 2 ] , the weights obtained by
Adam satisfy
[Equation]
where ∀ i ∈ [ d ] : u i = sign ( w (0)2 i ) ∈ {± 1 } and
[Equation]
(cid:12)(cid:12)
c ( t ) u
i (cid:12)(cid:12) ≤ δ.
Hence by Lemma 28, when t ∈ [ T Adam , 1 , T Adam , 2 ] , we have for 1 ≤ i ≤ d , (cid:113)(cid:80) j (cid:54) =
i (cid:0) H ( t )
[ i, j ] (cid:1)(cid:12)(cid:12) H ( t )
[ i, i ] (cid:12)(cid:12) ≤ 1 + δ 1 − δ (cid:32) 1 + (cid:12)(cid:12) c
( t ) (cid:12)(cid:12) (cid:107) v ( t ) (cid:107)(cid:33) (cid:115)(cid:80)
dj
=1 u 2 j u 2 i + (cid:13)(cid:13) E ( t ) (cid:13)(cid:13)(1 − δ )
2 u 2 i (cid:13)(cid:13) v ( t ) (cid:13)(cid:13)
2= O (cid:32) 1 + (cid:12)(cid:12) c ( t ) (cid:12)(cid:12) (cid:107) v ( t ) (cid:107)(cid:33) √ d + O (cid:32)(cid:13)(cid:13) E ( t ) (cid:13)(cid:13)(cid:13)(cid:13) v ( t ) (cid:13)(cid:13) 2(cid:33) , (51) and for 1 ≤ i ≤ d, 1 ≤ k ≤ d , (cid:113)(cid:80) j (cid:54) = id + k (cid:0) H ( t )
[ i, j ] (cid:1)(cid:12)(cid:12) H ( t )
[ id + k, id + k ] (cid:12)(cid:12) ≤ 1 + δ 1 − δ
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
( t
[Equation]
(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)
[Equation]
( t
[Equation]
(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
( t
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
(cid:0)
[Equation]
( t ) (cid:1)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)
[Equation]
[Equation]
(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
( t )
[Equation]
(cid:12)(cid:12)(cid:12) (cid:0)
[Equation]
( t
[Equation]
(cid:1)
[Equation]
Recall the following facts of Adam.
(A) By Lemma 14, we know that for t ∈ [ T Adam , 1 , T
(where T 1 is deﬁned
in
Deﬁnition
), w.h.p.
∀ k ∈
d
] : v (
t
) k
= c ( t ) =
η ( t − t inc ) .
Specially, when t = T Adam , 1 , ∀ k ∈ [ d ] : v ( t ) k
= c ( t ) = 1 d α 2 .
Lemma 24 and 26 tell us that for t ∈ [ T 1 , T Adam , 2 ] w.h.p.
∀ i, j ∈ [ d ] : | W 1 [ i, j ] | = ˜Θ (cid:16) 1 √ d (cid:17) , | w 2 i | = ˜Θ (cid:16) 1 √ d (cid:17) , which gives us ∀ k ∈ [ d ] : (cid:12)(cid:12)(cid:12) v ( t ) k (cid:12)(cid:12)(cid:12) = ˜Θ (cid:16) 1 √ d (cid:17) and (cid:12)(cid:12) c ( t ) (cid:12)(cid:12) = ˜Θ (cid:16) 1 √ d (cid:17) .
That means when t ∈ [ T Adam , 1 , T Adam , 2 ] ,
∀ k ∈
[ d ] : (cid:12)(cid:12)(cid:12) v ( t ) k (cid:12)(cid:12)(cid:12) and
(cid:12)(cid:12)
c
( t
) (cid:12)(cid:12) increase from d
α
2 to
˜Θ
√ d
) and
(cid:12)(cid:12)(cid:12)
v ( t ) k (cid:12)(cid:12)(cid:12)
| c (
t
) | = ˜Θ(1) , | c ( t ) | (cid:107) v t ) (cid:107) =
˜Θ (cid:16) 1 √ d (cid:17) .
(B) Lemma 14 and 25 tell us that w.h.p.
(cid:13)(cid:13)
[Equation] [Equation]
E
[Equation]
) decreases from Θ( d ) (resp. Θ(1) ) when t =
T Adam , 1 to ˜ O
(cid:0) d 2 √ η (cid:1) (resp.
˜ O (cid:0) d √ ηd (cid:1) )
when t = T Adam , 2 .
Combining (A) and (B), we get that the trend of (cid:107) E ( t ) (cid:107) 2 (cid:107) v ( t ) (cid:107) and (cid:12)(cid:12)(cid:12) E (
[Equation]
) k (cid:12)(cid:12)(cid:12) ( c (
[Equation]
) 2 is to decrease over time, and when t = T Adam , 2 , we have w.h.p.
[Equation]
Substituting (A) and eq.
(53) into eq.
(51) and (52) gives us w.h.p.,
[Equation]
where the trend of r ( t ) 1 i is to decrease over time and r ( T Adam , 2 )
i
[Equation]
We also have
[Equation]
where the trend of r ( t ) 2 i is to decrease over time and r ( T Adam , 2 )
i
[Equation]
Hence R Adamdiag ( t ) =
˜ O (cid:16) √ d
[Equation]
+ 1 d 2 + d
[Equation]
di
[Equation]
1 r ( t ) 1 i + d d 2 + d (cid:80) di =1 r ( t )
i :=
˜ O (cid:16) √ d (cid:17) + r ( t ) where the trend of r ( t ) is to decrease over time and
[Equation]
For any p > 0 , by picking the same hyperparameters as in Theorem 1, we have ηd 4 ≤ ˜ O (cid:0) 1 d p (cid:1) and hence r ( T Adam , 2 ) ≤ ˜ O
(cid:16)d p
[Equation] [Equation]
E. P ROOF OF L EMMA
By the assumed weight structure
[Equation]
we get that
[Equation] [Equation]
[Equation]
For the ( id + k ) -th row where 1 ≤ i ≤ d, 1 ≤ k ≤ d , i.e. the (( i − 1) d + k ) -th row of the submatrix [ H
H 11 ] , by triangle inequality again, we have (cid:115) (cid:88)
j (cid:54) = id + k
H 2 [ i, j ] ≤ (cid:115) (cid:88)
j (cid:54) =(
i − 1)
d
+ k
H
[( i − 1) d + k, j ] + (cid:118)(cid:117)(cid:117)(cid:116)
d (cid:88)
j =
H
[( i − 1) d + k, j ] ≤ (cid:115)(cid:88)
j (cid:54) =
i w
i w 22 j + (cid:118)(cid:117)(cid:117)(cid:116)
d (cid:88)
j = w 22 i W 21 [ j, k ] + | E k | = | w 2 i |  (cid:115)(cid:88)
j (cid:54) =
i w 22 j + (cid:118)(cid:117) (cid:117)(cid:116) d (cid:88)
j =
W 21 [ j, k ] 

+ | E k | .
Then we have that for 1 ≤ i ≤ d, 1 ≤ k ≤ d , (cid:113)(cid:80) j (cid:54) =
id + k H 2 [ i, j ] | H
[ id + k, id + k ] | ≤ | w 2 i | (cid:113)(cid:80)
j (cid:54) =
i w 22 j + (cid:113)(cid:80) dj
=1 W 21 [ j, k ] w 22 i + | E k | w 22 i = (cid:115)(cid:80)
j (cid:54) =
i w
j w 2 2 i + (cid:115)(cid:80) dj
=1 W 21 [ j, k ] w 2 2 i + | E k | w 2 2 i ≤ (cid:115) (1 + δ )(1 − δ ) 2 · (cid:80)
j (cid:54) =
i c 2 u 2 j c 2 u 2 i + (cid:115) (1 + δ )(1 − δ ) 2 · v 2 k (cid:80) dj
=1 u 2 j c
[Equation]
u 2 i
[Equation]
|
[Equation]
k | (
[Equation]
δ )
[Equation]
c 2 u 2 i ≤ 1 + δ 1 − δ (cid:18)
[Equation]
| v
[Equation]
|
[Equation]
c | (cid:19) (cid:115)(cid:80) dj
[Equation]
u
[Equation]
|
[Equation]
| (
[Equation] [Equation]
F C ONNECTION BETWEEN DIAGONAL OF LOSS H ESSIAN AND WEIGHTS
The partial derivative at W i of the cost function for each i is given by:
[Equation]
In our experiments
[Equation]
we were interested in the diagonal elements of the hessian.
These are given by:
[Equation]
for each possible i, a, b .
For ease in notation, deﬁne for each i , the quantities M i :=
W Ti +1 . . .
W TH + and N i :=
W T 1 . . .
W Ti − 1 .
Then we have the following lemma.
Lemma 29.
The diagonal elements of the hessian of the cost function are given by:
[Equation]
for each possible i, a, b .
Proof.
We have:
[Equation]
This implies that:
[Equation]
where the last step follows since M i and N i are not functions of W i .
Since M i :=
W Ti +1 . . .
W TH +1 , N i :=
W T 1 . . .
W Ti − 1 , by deﬁning C i :=
M i W H +1 W H . . .
W i +1 =
M i M Ti and D i :=
W i − 1 . . .
W 2 W 1 N i =
N Ti
N
i
we have that:
[Equation]
where C i and D i are not functions of W i .
Now, Equation 74 in the Matrix Cookbook 12 shows us that for any matrices A and X we have:
[Equation]
Note that W i ∈ R
d i × d i − 1 , then we can apply this to obtain that:
[Equation]
This completes the proof.
For ease of notation, let’s now drop the superscript OPT and ( t ) and write R OPTmed , 1 ( t ) as R med , 1 and
R OPTmed , 2 ( t ) as R med , 2 .
For a 2-layer linear network, H = 1 . Consider the Hessian w.r.t W 1 , we have
M 1 M T 1 = W T 2 W 2 and N T 1 N 1 is an identity matrix.
Under Assumption 1, we know that W 2 is a row vector, which can be denoted as W 2 =
[ w 21 , w 22 , ..., w 2 d 1 ] .
Then we have
[Equation]
Similarly, consider the Hessian w.r.t.
W 2 , we have that M 1 M T 1 is an identity matrix and N T 1 N 1 =
W 1 W T 1 .
Therefore,
[Equation]
Hence we have related the uniformity of diagonal Hessian to that of weight matrices.
In the detailed analysis, for both GD and Adam, we can prove that W 1 converges to an approximately rank 1 matrix.
The following lemma allows us to use this rank 1 structure to compute R med , 1 and R med , 2 .
Lemma 30.
Suppose W 1 ∈ R d × d and W 2 ∈ R 1 × d have the following structure:
[Equation]
where u ∈ R d , v ∈ R d , R 1 ∈ R d × d , R 2 ∈ R 1 × d and that ∀ 1 ≤ i, j ≤ d : | R 1 [
[Equation]
, j
[Equation]
| | u i v j | ≤ δ, | R 2 i | | cu i | ≤ δ, δ ∈ (0 , 1) .
Then we have
R med , 1 , R med , 2 ∈ (cid:20) (1 − δ )(1 + δ ) 2 · max i u
i median u 2 i , (1 + δ )(1 − δ ) 2 · max
i
u
i median u
i (cid:21) .
Proof.
Let’s ﬁrst consider R med , 1 .
we have
[Equation] [Equation]
δ )
[Equation]
median ( cu i ) 2 ≤ median w 22 i ≤ (1 + δ ) 2 median ( cu i ) 2 , which yields (1 − δ )(1 + δ ) 2 · max i u
i median u 2 i ≤ R med , 1 = max i
w
i median w 22 i ≤ (1 + δ )(1 − δ ) 2 · max i u
i median u 2 i .
Similarly, for R med , 2 .
We have that
[Equation]
⇒
[Equation]
u
[Equation]
which yields
[Equation]
That means
[Equation]
G A UXILIARY LEMMAS
Lemma 31.
Let A = 1 m Y X T , Λ xx :
= 1 m XX T , g ( t ) k
= ∇ W k L ( W ( t ) ) , k = 1 , 2 .
Denote ˜ A ( t ) , ˜Λ ( t ) xx and ˜ g ( t ) k , k
= 1 , 2 as the corresponding batch versions at time t .
Let M ( t ) 1 = max
i,j (cid:12)(cid:12)(cid:12) W ( t ) 1 [ i, j ] (cid:12)(cid:12)(cid:12) and
M ( t ) 2 = max i (cid:12)(cid:12)(cid:12) w ( t ) 2 i (cid:12)(cid:12)(cid:12) .
Under Assumption 3, we have with probability at least 1 − 1 d , for ∀ t ≤ T and ∀ i, j ∈ [ d ] ,
[Equation]
+
[Equation]
t
[Equation]
d 2 T.
Proof.
By Assumption 3 and Chebyshev’s inequality, we have for ﬁxed i, j ∈ [ d ] and t ≤ T ,
[Equation]
Applying the union bound gives us
[Equation]
which gives us with probability at least 1 − 1 d , for ∀ t ≤ T, ∀ i, j ∈
[ d ] ,
[Equation]
Now we are ready to bound ˜ g
( t ) k − g ( t ) k for k = 1 , 2 and t ≤ T .
Note that for all t ≤ T and ∀
i ∈
[ d ] ,
[Equation]
] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ d (cid:88)
j = (cid:12)(cid:12)(cid:12) w ( t ) 2 j (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) W ( t ) 1 [ j, i ] (cid:12)(cid:12)(cid:12) ≤ dM ( t ) 1 M ( t ) 2 .
Then we have with probability at least 1 − 1 d , for all t ≤ T and ∀
i ∈
[ d ] , (cid:12)(cid:12)(cid:12)(cid:16) W ( t ) 2 W ( t ) 1 (cid:16) ˜Λ ( t ) xx − Λ xx (cid:17)(cid:17) i (cid:12)(cid:12)(cid:12)
≤ d (cid:88)
j = (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) W ( t ) 2 W ( t ) 1 (cid:17) j (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) ˜Λ ( t ) xx
[ j, i ] − Λ xx
[ j, i ] (cid:12)(cid:12)(cid:12)
[Equation]
Combining with ˜ g
( t ) 1 − g
( t ) 1 = W ( t ) T(cid:16)
W ( t ) 2 W ( t ) 1 (cid:16) ˜Λ ( t ) xx − Λ xx (cid:17) − (cid:16) ˜
A ( t ) − A (cid:17)(cid:17) , we get that with probability at least 1 − 1 d , for all t ≤ T and ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) ˜
g ( t )
[ i, j ] − g
( t )
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Similarly, note that ˜ g ( t ) 2 i − g ( t ) 2 i = (cid:16) W ( t ) 2 W ( t ) 1 (cid:16) ˜Λ ( t ) xx − Λ xx (cid:17) − (cid:16) ˜
A ( t ) − A (cid:17)(cid:17) W ( t ) T, we then have that with probability at least 1 − 1 d , for all t ≤ T and ∀ i, j ∈ [ d ] , (cid:12)(cid:12)(cid:12) ˜
g ( t )
i − g
( t )
i (cid:12)(cid:12)(cid:12) ≤ d (cid:88)
j = (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) W ( t ) 2 W ( t ) 1 (cid:16) ˜Λ ( t )
xx − Λ xx
[Table]
j
[Table]
[Equation]
Lemma 32.
Consider two sequences { a ( t ) } t ≥ 0 , { b ( t ) } t ≥ 0 , which satisfy a ( t ) =
(1 − β ) t (cid:88) τ =
[Equation]
∈ (
[Equation]
Suppose ∀ τ ≤ t : (cid:12)(cid:12) b ( t ) (cid:12)(cid:12) ≤ B , then for any (cid:15) > 0 , the following truncated version ˜ a ( t ) =
(1 − β )
H (cid:88) τ = β τ b ( t − τ ) with H ≥ 1 1 − β log B(cid:15)
= ˜Ω (cid:16) 1 1 − β (cid:17) satisﬁes
[Equation]
Proof.
We have that
[Equation]
To make it less than (cid:15) , it sufﬁces to choose H ≥ log( (cid:15)B ) / log β .
Since β ∈ (0 , 1) , we know that log β ≤ β − 1 < 0 .
We also have log (cid:15)B < 0 .
Then it sufﬁces to choose
H ≥ log( (cid:15)/B ) β
−≥ log( (cid:15)/B ) log β ⇒
H ≥ 1 − β log
B (cid:15) = ˜Ω (cid:18) 1 − β (cid:19) .
Lemma 33.
Suppose a, b, c, e a , e b , e c ∈ R , b > 0 , c > 0 satisfy b + e b
+
e c > 0 , | e a | ≤ δ | a | , | e b | ≤ δb, | e c | ≤ δ 2 c 2 with 0 < δ (cid:28) 1 , then we have a + e
[Equation]
√
[Equation]
b + c (1 + R )
[Equation]
where | R | =
O ( δ ) .
Proof.
We have a
[Equation]

[Equation]
Deﬁne R :=
q 1 + q 2 .
The term | q 1 | can be bounded by | q 1 | = (cid:12)(cid:12)(cid:12)
[Equation]
≤ | e
[Equation]
+ | e c | ( √ b + e b + e c + c ) (cid:16) √ b + √ b + e b + e c (cid:17) ≤ | e b | ( √ b + e b + e c + c ) √ b + (cid:112) | e c | c · (cid:112) | e c | √ b + √ b + e b + e c ≤ | e b | ( √ b + e b + e c + c ) √ b (cid:124) (cid:123)(cid:122) (cid:125) q+ δ (cid:112) | e c | √ b + √ b + e b + e c
(cid:124) (cid:123)(cid:122) (cid:125) q, where | q 3 | can be bounded by |
[Equation]
|
[Equation]
i
[Equation]
Here the denominator of ( i ) uses b + e b ≥ b (1 − δ ) > 0 and √ x + y ≥ √ x − (cid:112) | y | when x ≥ 0 , x + y ≥ 0 .
Now let’s bound | q 4 | .
If e c > 0 , we have e c = | e c | and | q 4 | ≤ √ e c √ e c = 1 since b + e b ≥ b (1 − δ ) > 0 .
If e c ≤ 0 , note that b + e b + e c > 0 , we have | e c | < b + b e ≤ b (1 + δ ) , which yields | q 4 | ≤ √ | e c | √ b
[Equation]
O (1) .
Combining the above bounds give us | q 1 | ≤ | q 3 | + δ | q 4 | =
O ( δ ) .
On the other hand, | q 2 | can be bounded by
[Equation]
Then | R | ≤ | q 1 | + | q 2 | =
O ( δ )
Lemma 34.
Suppose X 1 , X 2 , ..., X d are i.i.d Gaussian with mean 0 and variance σ 2 , then for 0 < δ < 1 e , we have with probability at least 1 − δ ,
[Equation]
i
[Equation]
for some C 1 , C 2 > 0 .
Proof.
It sufﬁces to assume that σ 2 = 1 and prove that w.p. at least 1 − δ , max 1 ≤ i ≤ d
X 2 i ≥
C 1 log d
[Equation]
C
[Equation]
log log 1 δ .
First, by the lower bound of Gaussian tail, there exists α, β > 0 such that P ( | X i | > x ) = 2 P ( X i > x ) ≥ αe − βx 2 for x ≥ 0 .
Then by i.i.d., we have
[Equation]
where the last inequality uses 1 − x ≤ e − x for x ∈ [0 , 1] .
Let exp( − dαe − βx 2 )
=
δ , we get that w.p.
at least 1 − δ ,
[Equation]
Then we have w.p. at least 1 − δ , max
[Equation]
Lemma 35.
Suppose X 1 , X 2 , ..., X d are i.i.d Gaussian with mean 0 and variance σ 2 , then we have with constant probability,
[Equation]
Proof.
It sufﬁces to assume that σ 2 = 1 and prove that with constant probability, 1 d (cid:80) di =1 1 | X i | ≤
O (log d ) .
Consider X i for some ﬁxed i .
Since X i ∼ N (0 , 1) , we have P ( | X i | ≤ t ) ≤ 2 t √ 2 π .
Then we know that with probability at least 1 − Θ (cid:0) d − 1 (cid:1) , | X i | ≥ Cd for some C > 0 .
Then by union bound, with constant probability, ∀ i ∈ [ d ] : | X i | ≥ Cd .
Now we split the interval [ Cd , 1] into several subintervals
I k
= { i : | X i | ∈
[2 − k − 1 , 2 − k ] } for k = 0 , 1 , ..., (cid:100) log 2 dC (cid:101) − 1 .
Let p
k
=
P
( | X i | ∈
[2 − k − 1 , 2 − k ]) , we know that |I k | ∼ Binomial ( d, p k ) and p k ≤ C 1 · 2 − k − 1 .
Then by the concentration of binomial variables, we have w.p. at least 1 − d − p for p > 0 ,
[Equation]
I
[Equation]
Then we have (cid:88) i ∈I k
[Equation]
Therefore, with constant probability,
[Equation]
which means with constant probability, 1 d (cid:80) di =1 1 |
X i | =
O (log d )
[Equation]