C ONVERGENCE RATE OF SIGN STOCHASTIC GRADIENT DESCENT FOR NON - CONVEX FUNCTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
The sign stochastic gradient descent method (signSGD) utilizes only the sign of the stochastic gradient in its updates.
Since signSGD carries out one-bit quantization of the gradients, it is extremely practical for distributed optimization where gradients need to be aggregated from different processors.
For the ﬁrst time, we establish convergence rates for signSGD on general non-convex functions under transparent conditions.
We show that the rate of signSGD to reach ﬁrst-order critical points matches that of SGD in terms of number of stochastic gradient calls, up to roughly a linear factor in the dimension.
We carry out simple experiments to explore the behaviour of sign gradient descent (without the stochasticity) close to saddle points and show that it often helps completely avoid saddle points without using either stochasticity or curvature information.
I NTRODUCTION
Deep neural network training takes place in an error landscape that is high-dimensional, non-convex and stochastic.
In practice, simple optimization techniques perform surprisingly well but have very limited theoretical understanding.
While stochastic gradient descent (SGD) is widely used, algorithms like Adam (Kingma & Ba, 2015), RMSprop (Tieleman & Hinton, 2012) and Rprop (Riedmiller & Braun, 1993) are also popular.
These latter algorithms involve component-wise rescaling of gradients, and so bear closer relation to signSGD than SGD.
Currently, convergence rates have only been derived for close variants of SGD for general non-convex functions, and indeed the Adam paper gives convex theory.
Recently, another class of optimization algorithms has emerged which also pays attention to the resource requirements for training, in addition to obtaining good performance.
Primarily, they focus on reducing costs for communicating gradients across different machines in a distributed training environment (Seide et al., 2014; Strom, 2015;
Li et al., 2016;
Alistarh et al., 2017; Wen et al., 2017).
Often, the techniques involve quantizing the stochastic gradients at radically low numerical precision.
Empirically, it was demonstrated that one can get away with using only one-bit per dimension without losing much accuracy (Seide et al., 2014; Strom, 2015).
The theoretical properties of these approaches are however not well-understood.
In particular, it was not known until now how quickly signSGD
(the simplest incarnation of one-bit SGD) converges or even whether it converges at all to the neighborhood of a meaningful solution.
Our contribution : we supply the non-convex rate of convergence to ﬁrst order critical points for signSGD.
The algorithm updates parameter vector x k according to
[Equation]
where ¯ g k is the mini-batch stochastic gradient and δ k is the learning rate.
We show that for nonconvex problems, signSGD entertains convergence rates as good as SGD, up to a linear factor in the dimension.
Our statements impose a particular learning rate and mini-batch schedule.
Ours is the ﬁrst work to provide non-convex convergence rates for a biased quantisation procedure as far as we know, and therefore does not require the randomisation that other gradient quantisation algorithms need to ensure unbiasedness.
The technical challenge we overcome is in showing how to carry the stochasticity in the gradient through the sign non-linearity of the algorithm in a controlledfashion.
Whilst our analysis is for ﬁrst order critical points, we experimentally test the performance of sign gradient descent without stochasticity (signGD) around saddle points.
We removed stochasticity in order to investigate whether signGD has an inherent ability to escape saddle points, which would suggest superiority over gradient descent (GD) which can take exponential time to escape saddle points if it gets too close to them (Du et al., 2017).
In our work we make three assumptions.
Informally, we assume that the objective function is lowerbounded, smooth, and that the stochastic gradient has tails that vanish like a Gaussian distribution or faster.
These assumptions are very general and hold for a wide class of functions, including the ones encountered in deep learning.
Outline of paper : in Section 3 we formally state our assumptions.
In Section 4 we give our analysis of signSGD.
In Section 5 we compare our results to SGD.
In Section 6 we experimentally test the ability of the signGD (without the S) to escape saddle points.
R ELATED WORK
Deep learning: the prototypical optimisation algorithm for neural networks is stochastic gradient descent (SGD)—see Algorithm 2.
The deep learning community has discovered many practical tweaks to ease the training of large neural network models.
In Rprop (Riedmiller & Braun, 1993) each weight update ignores the magnitude of the gradient and pays attention only to the sign, bringing it close to signSGD.
It differs in that the learning rate for each component is modiﬁed depending on the consistency of the sign of consecutive steps.
RMSprop (Tieleman & Hinton, 2012) is Rprop adapted for the minibatch setting—instead of dividing each component of the gradient by its magnitude, they estimate the rescaling factor as an average over recent iterates.
Adam (Kingma & Ba, 2015) is RMSprop with momentum, meaning both gradient and gradient rescaling factors are estimated as bias-corrected averages over iterates.
Indeed switching off the averaging in Adam yields signSGD.
These algorithms have been applied to a breadth of interesting practical problems, e.g.
(Xu et al., 2015; Gregor et al., 2015).
Some hope that having a more principled understanding of why deep learning works could lead to improved algorithms as well as a general feeling of satisfaction.
To this end, Dauphin et al.
(2014) frame the primary obstacle to neural network training as the proliferation of saddle points in high dimensional objectives.
Practitioners challenge this view, suggesting that saddle points may be seldom encountered at least in retrospectively successful applications of deep learning (Goodfellow et al., 2015).
Optimisation theory: in convex optimisation there is a natural notion of success—rate of convergence to the global minimum x ∗ .
Convex optimisation is eased by the fact that local information in the gradient provides global information about the direction towards the minimum, i.e. ∇ f ( x ) tells you information about x ∗ − x .
In non-convex problems ﬁnding the global minimum is in general intractable, so theorists usually settle for measuring some restricted notion of success, such as rate of convergence to stationary points (e.g. Allen-Zhu (2017a)) or local minima (e.g. Nesterov & Polyak (2006)).
Given the importance placed by Dauphin et al.
upon evading saddle points, recent work considers the efﬁcient use of noise (Jin et al., 2017; Levy, 2016) and curvature information (Allen-Zhu, 2017b) to escape saddle points and ﬁnd local minima.
Distributed machine learning: whilst Rprop and Adam were proposed by asking how we can use gradient information to make better optimisation steps, another school asks how much information can we throw away from the gradient and still converge at all.
Seide et al.
Strom (2015) demonstrated empirically that one-bit quantisation can still give good performance whilst dramatically reducing gradient communication costs in distributed systems.
Convergence of quantized stochastic gradient methods remains largely unknown.
Alistarh et al.
(2017) provide convergence rates for quantisation schemes that are unbiased estimators of the true gradient, and are thus able to rely upon vanilla SGD convergence results.
Wen et al.
(2017) prove asymptotic convergence of a {− 1 , 0 , 1 } ternary quantization scheme that also retains the unbiasedness of the stochastic gradient.
Our proposed approach is different, in that we directly employ the sign gradient which is biased .
This avoids the randomization needed for constructing an unbiased quantized estimate.
To the best of our knowledge, the current work is the ﬁrst to establish a convergence rate for a biased quantisation scheme, and our proof differs to that of vanilla SGD.
The fact that signSGD is related to both attempts to improve gradient descent like Rprop and Adam, and attempts to damage it but not too badly like quantised SGD is intriguing.
A SSUMPTIONS
Assumption 1 (The objective function is bounded below) .
For all x and some constant f ∗ , the objective function satisﬁes
[Equation]
Remark: this assumption applies to every practical objective function that we are aware of.
Assumption 2 (The objective function is L-Lipschitz smooth) .
Let g ( x ) denote the gradient of the objective f ( . ) evaluated at point x .
Then for every y we assume that
[Equation]
Remark: this assumption allows us to measure the error in trusting the local linearisation of our objective, which will be useful for bounding the error in a single step of the algorithm.
For signSGD we can actually relax this assumption to only hold only for y within a local neighbourhood of x , since signSGD takes steps of bounded size.
Assumption 3 (The noise in the stochastic gradient has vanishing tails) .
In particular, we assume that the mini-batch stochastic gradient satisﬁes Hoeffding’s inequality with variance-proxy σ .
Consider a stochastic gradient estimate averaged over a mini-batch of size n .
[Equation]
Let ¯ g i ( x ) refer to the i th component of ¯ g ( x ) .
And let g ( x ) refer to the true gradient at x .
Then we assume that there exists a parameter σ such that the tails of the distribution of ¯ g i ( x ) decay like
[Equation]
Remark: this assumption encodes the natural idea that increasing mini-batch size should reduce variance in the stochastic gradient estimate.
It holds for noise distributions that are Gaussian or sub-Gaussian.
N ON - CONVEX CONVERGENCE RATE OF SIGNSGD
Informally, our primary result says that if we run signSGD with the prescribed learning rate and mini-batch schedules, then after N stochastic gradient evaluations, we should expect that somewhere along the optimisation trajectory will be a place with gradient 1-norm smaller than O( N − 0 . 25 ).
This matches the non-convex SGD rate, insofar as they can be compared.
Before we dive into the theorems, here’s a refresher on our notation—deep breath— g k is the gradient at step k , f ∗ is the lower bound on the objective function, f 0 is the initial value of the objective function, d is the dimension of the space, K is the total number of iterations, N K is the cumulative number of stochastic gradient calls at step K , σ is the intrinsic variance-proxy for each component of the stochastic gradient, and ﬁnally L is the maximum curvature (see Assumption 2).
1: Inputs : x 0 , K (cid:46) initial point and time budget 2: for k ∈ [0 , K − 1] do 3: δ k ← learningRate ( k ) 4: n k ← miniBatchSize ( k ) 5: ¯ g k ← 1 n k (cid:80) n k i =
1 stochasticGradient ( x k ) 6: x k +1 ← x k − δ k sign (¯ g k ) (cid:46) the sign operation is element-wise
1: Inputs : x 0 , K (cid:46) initial point and time budget 2: for k ∈ [0 , K − 1] do 3: δ k ← learningRate ( k ) 4: n k ← miniBatchSize ( k ) 5:
ˆ g k ← 1 n k (cid:80) n k i =1 stochasticGradient ( x k ) 6: x k +1 ← x k − δ k
¯ g k
Theorem 1 (Non-convex convergence rate of signSGD) .
Apply Algorithm 1 under Assumptions 1, 2 and 3.
Schedule the learning rate and mini-batch size as
[Equation]
Let N K be the cumulative number of stochastic gradient calls up to step K , i.e. N K
= O
( K 2 )
Then we have
[Equation]
Theorem 2 (Non-convex convergence rate of stochastic gradient descent) .
Apply Algorithmunder Assumptions 1, 2 and 3.
Schedule the learning rate and mini-batch size as
[Equation]
Let N K be the cumulative number of stochastic gradient calls up to step K , i.e. N K = K .
Then we have that
[Equation]
The proofs are deferred to Appendix A and here we sketch the intuition for Theorem 1.
First consider the non-stochastic case: we know that if we take lots of steps for which the gradient is large, we will make lots of progress downhill.
But since the objective function has a lower bound, it is impossible to keep taking large gradient steps downhill indeﬁnitely, therefore increasing the number of steps requires that we must run into somewhere with small gradient.
To get a handle on this analytically, we must bound the per-step improvement in terms of the norm of the gradient.
Assumption 2 allows us to do exactly this.
Then we know that the sum of the per-step improvements over all steps must be smaller than the total possible improvement, and that gives us a bound on how large the minimum gradient that we see can be.
In the non-stochastic case, the obstacle to this process is curvature.
Curvature means that if we take too large a step the gradient becomes unreliable, and we might move uphill instead of downhill.
Since the step size in signSGD is set purely by the learning rate, this means we must anneal the learning rate if we wish to be sure to control the curvature-induced error and make good progress downhill.
Stochasticity also poses a problem in signSGD.
In regions where the gradient signal is smaller than the noise, the noise is enough to ﬂip the sign of the gradient.
This is more severe than the additive noise in SGD, and so the batch size must be grown to control this effect.
You might expect that growing the batch size should lead to a worse convergence rate than SGD.
This is forgetting that signSGD has an advantage in that it takes large steps even when the gradient is small.
It turns out that this positive effect cancels out the fact that the batch size needs to grow, and the convergence rate ends up being the same as SGD.
For completeness, we also present the convergence rate for SGD derived under our assumptions.
The proof is given in Appendix B. Note that this appears to be a classic result, although we are not sure of the earliest reference.
Authors often hide the dimension dependence of the variance bound.
SGD does not require an increasing batch size since the effect of the noise is second order in the learning rate, and therefore gets squashed as the learning rate decays.
The rate ends up being the same in N K as signSGD because SGD makes slower progress when the gradient is small.
C OMPARING THE CONVERGENCE RATE TO SGD
To make a clean comparison, let us set δ = 1 L (as is often recommended) and hide all numerical constants in Theorems 1 and 2.
Then for signSGD, we get
[Equation]
and for SGD we get
[Equation]
where ∼ denotes general scaling.
What do these bounds mean?
They say that after we have made a cumulative number of stochastic gradient evaluations N , that we should expect somewhere along our trajectory to have hit a point with gradient norm smaller than N − 14 .
One important remark should be made.
SignSGD more naturally deals with the one norm of the gradient vector, hence we had to square the bound to enable direct comparison with SGD 1 .
This means that the constant factor in signSGD is roughly worse by a square.
Paying attention only to dimension, this looks like
[Equation]
This defect in dimensionality should be expected in the bound, since signSGD almost never takes the direction of steepest descent, and the direction only gets worse as dimensionality grows.
This raises the question, why do algorithms like Adam, which closely resemble signSGD, work well in practice?
We suggest that it may be because they take advantage of additional structure in neural network landscapes that is not captured by our relatively straightforward assumptions.
We will broach this subject more fully in the discussion section.
S WINGING BY SADDLE POINTS ? A N EXPERIMENT
Seeing as our theoretical analysis only deals with convergence to stationary points, it does not address how signSGD might behave around saddle points.
We wanted to investigate the na¨ıve intuition that gradient rescaling should help ﬂee saddle points—or in the words of Zeyuan Allen-Zhu—swing by them.
For a testbed, the authors of (Du et al., 2017) kindly provided their 10-dimensional ‘tube’ function.
The tube is a specially arranged gauntlet of saddle points, each with only one escape direction, that must be navigated in sequence before reaching the global minimum of the objective.
The tube was designed to demonstrate how stochasticity can help escape saddles.
Gradient descent takes much longer to navigate the tube than perturbed gradient descent of (Jin et al., 2017).
It is interesting to ask, even empirically, whether the sign non-linearity in signSGD can also help escape saddle points efﬁciently.
For this reason we strip out the stochasticity and pit the sign gradient descent method (signGD) against the tube function.
There are good reasons to expect that signGD might help escape saddles—for one, it takes large steps even when the gradient is small, which could drive the method away from regions of small gradient.
For another, it is able to move in directions orthogonal to the gradient, which might help discover escape directions of the saddle.
We phrase this as signGD’s greater ability to explore .
Our experiments revealed that these intuitions sometimes hold out, but there are cases where they break down.
In Figure 1, we compare the sign gradient method against gradient descent, perturbed gradient descent (Jin et al., 2017) and rescaled gradient descent (cid:16) x k +
=
x
k − g (cid:107) g
(cid:107) 2 (cid:17) which is a noiseless version of the algorithm considered in (Levy, 2016).
No learning rate tuning was conducted, so we suggest paying attention to the qualitative behaviour rather than the ultimate convergence speed.
The left hand plot pits the algorithms against the vanilla tube function.
SignGD has very different qualitative behaviour to the other algorithms—it appears to make progress completely unimpeded by the saddles.
We showed that this behaviour is partly due to the axis alignment of the tube function, since after randomly rotating the objective the behaviour changes (although it is still qualitatively different to the other algorithms).
One unexpected result was that for certain random rotations of the objective, signGD could get stuck at saddle points (see right panel in Figure 1).
On closer inspection, we found that the algorithm was getting stuck in perfect periodic orbits around the saddle.
Since the update is given by the learning rate multiplied by a binary vector, if the learning rate is constant it is perfectly possible for a sequence of updates to sum to zero.
We expect that this behaviour relies on a remarkable structure in both the tube function and the algorithm.
We hypothesise that for higher dimensional objectives and a non-ﬁxed learning rate, this phenomenon might become extremely unlikely.
This seems like a worthy direction of future research.
Indeed we found empirically that introducing momentum into the update rule was enough to break the symmetry and avoid this periodic behaviour.
D ISCUSSION
First we wish to discuss the connections between signSGD and Adam (Kingma & Ba, 2015).
Note that setting the Adam hyperparameters β 1 = β 2 = (cid:15) = 0 , Adam and signSGD are equivalent.
Indeed the authors of the Adam paper suggest that during optimisation the Adam step will commonly look like a binary vector of ± 1 (multiplied by the learning rate) and thus resemble the sign gradient step.
If this algorithmic correspondence is valid, then there seems to be a discrepancy between our theoretical results and the empirical good performance of Adam.
Our convergence rates suggest that signSGD should be worse than SGD by roughly a factor of dimension d .
In deep neural network applications d can easily be larger than 10 6 .
We suggest a resolution to this proposed discrepancy— there is structure present in deep neural network error surfaces that is not captured by our simplistic theoretical assumptions.
For example it is reasonable to expect that neural network error surfaces might exhibit only weak coupling across dimensions.
To provide intuition for how such an assumption can help improve the dimension scaling of signSGD, note that in the idealised case of total decoupling (the Hessian is everywhere diagonal) then the problem separates into d independent one dimensional problems, so the dimension dependence is lost.
Next, let’s talk about saddle points.
Though general non-convex functions are littered with local minima, recent work rather characterises successful optimisation as the evasion of a web of saddle points (Dauphin et al., 2014).
Current theoretical work focuses either on using noise Levy (2016); Jin et al.
or curvature information (Allen-Zhu, 2017b) to establish bounds on the amount of time needed to escape saddle points.
We noted that merely passing the gradient through the sign operation introduces an algorithmic instability close to saddle points, and we wanted to empirically investigate whether this could be enough to escape them.
We removed stochasticity from the algorithm to focus purely on the effect of the sign function.
We found that when the objective function was axis aligned, then sign gradient descent without stochasticity (signGD) made progress unhindered by the saddles.
We suggest that this is because signGD has a greater ability to ‘explore’, meaning it typically takes larger steps in regions of small gradient than SGD, and it can take steps almost orthogonal to the true gradient direction.
This exploration ability could potentially allow it to break out of subspaces convergent on saddle points without sacriﬁcing its convergence rate—we hypothesise that this may contribute to the often superior practical performance of algorithms like Rprop and Adam, which bear closer relation to signSGD than
SGD.
For non axis-aligned objectives, signGD could sometimes get stuck in perfect periodic orbits around saddle points, though we hypothesise that this behaviour may be much less likely for higher dimensional objectives (the testbed function had dimension 10) with non-constant learning rate.
Finally we want to discuss the implications of our results for gradient quantisation schemes.
Whilst we do not analyse the multi-machine case of distributed optimisation, we imagine that our results will extend naturally to that setting.
In particular our results stand as a proof of concept that we can provide guarantees for biased gradient quantisation schemes.
Existing quantisation schemes with guarantees require delicate randomisation to ensure unbiasedness.
If a scheme as simple as ours can yield provable guarantees on convergence, then there is a hope that exploring further down this avenue can yield new and useful practical quantisation algorithms.
E XPERIMENTS
In this section, we investigate the numerical properties of variants of signSGD on training deep learning models.
Speciﬁcally, we train a convolutional neural networks (CNN) (Krizhevsky et al., 2012) on the MNIST data set and a version of residual network (ResNet18)
(He et al., 2016) on the
CIFAR-10 data set.
The point of the experiments is not to outperform state-of-the-art results but rather to compare the relative pros and cons of using the signSGD and understanding the impact of such 1-bit gradient compression relative to SGD under the same parameter conﬁgurations.
For MNIST, we used a ﬁxed learning rate of 0 . 01 for all algorithms except for signSGD with parameter chosen by Theorem 1, where we started with δ = 0 .
5 and decreased it by δ/ √
i at iteration i .
For CIFAR-10, we used initial learning rate of 0 . 05 and divided it by 10 every 25 epochs, an L regularisation of 4 × 10 − 5 and no data augmentation.
These conﬁguration has been tested to be quite effective and favorable for SGD on the two data sets respectively.
Our results (shown in Figure 2) suggest that the version of signSGD with the same learning rate as SGD schedule is able to attain very similar test accuracy on both data sets while the version with theoretical guarantee works reasonably well on MNIST but overﬁts signiﬁcantly on CIFAR10.
On the other hand, by inspecting the convergence plot (bottom leftmost) for CIFAR-10, it appears that SGD got stuck at a suboptimal region while signSGD was able to make further progress and converge to a region with a much smaller objective function.
In summary, the experimental results illustrate that SignSGD is a very practical algorithm that reduces the communication cost with almost no loss in accuracy relative to SGD.
In addition, echoing
[Figure]
our discussion earlier
[Figure]
SignSGD seems to be better than SGD in “exploration” and can often lead to a better local minima on real data sets too.
C ONCLUSION
We have investigated the theoretical properties of the sign stochastic gradient method (signSGD) as an algorithm for non-convex optimisation.
The study was motivated by links that the method has both to deep learning stalwarts like Adam and Rprop, as well as to newer quantisation algorithms that intend to cheapen the cost of gradient communication in distributed machine learning.
We have proved non-convex convergence rates for signSGD to ﬁrst order critical points.
Insofar as the rates can directly be compared, they are of the same order as SGD in terms of number of gradient evaluations, but worse by a linear factor in dimension.
SignSGD has the advantage over existing gradient quantisation schemes with provable guarantees, in that it doesn’t need to employ randomisation tricks to remove bias from the quantised gradient.
We wish to propose some intriguing directions for future work.
First our analysis only looks at convergence to ﬁrst order critical points.
Whilst we present preliminary experiments exhibiting success and failure modes of the algorithm around saddle points, a more detailed study attempting to pin down exactly when we can expect signSGD to escape saddle points efﬁciently would be welcome.
This is an interesting direction seeing as existing work always relies on either stochasticity or second order curvature information to avoid saddles.
Second the link that signSGD has to both Adam-like algorithms and gradient quantisation schemes is enticing.
In future work we intend to investigate whether this connection can be exploited to develop large scale machine learning algorithms that get the best of both worlds in terms of optimisation speed and communication efﬁciency.
A P ROVING THE CONVERGENCE RATE OF THE SIGN GRADIENT METHOD
Theorem 1 (Non-convex convergence rate of signSGD) .
Apply Algorithm 1 under Assumptions 1, 2 and 3.
Schedule the learning rate and mini-batch size as
[Equation]
Let N K be the cumulative number of stochastic gradient calls up to step K , i.e. N K
= O
( K 2 )
Then we have
[Equation]
Proof.
Our general strategy will be to show that the expected objective improvement at each step will be good enough to guarantee a convergence rate in expectation.
First let’s bound the improvement of the objective during a single step of the algorithm for one instantiation of the noise.
Note that I [ . ] is the indicator function, and g k,i denotes the i th component of the vector g k .
First use Assumption 2, plug in the step from Algorithm 1, and decompose the improvement to expose the stochasticity-induced error:
[Equation]
The important observation is that when the true gradient is larger than the standard deviation of the noise, the sign will typically be correct.
Mistakes will typically only be made when | g k,i | is smaller than σ .
This allows us to bound the error by σ .
Let’s make this intuition rigorous using Hoeffding’s inequality (Assumption 3).
Find the expected improvement at time k + 1 conditioned on the previous iterates.
Note that P [ E ] denotes the probability of event E .
[Equation]
Now use the fact that x exp( − x 2 ) <
[Equation]
and our schedules on the learning rate and batch size to write
[Equation]
Now take the expectation over the noise in all previous iterates, and sum over k :
[Equation]
Assumption
[Equation]
telescope
[Equation]
previous result
[Equation]
harmonic sum
We can rearrange this inequality to yield a rate:
[Equation]
Finally since we are growing our mini-batch size, it will take N K − 1 =
K ( K +
[Equation]
gradient evaluations to reach step K − 1 .
Using that 2 N K − 2 ≤ K 2 ≤ 2 N K − 1 yields the result.
B P ROVING THE CONVERGENCE RATE OF STOCHASTIC GRADIENT DESCENT
Theorem 2 (Non-convex convergence rate of stochastic gradient descent) .
Apply Algorithmunder Assumptions 1, 2 and 3.
Schedule the learning rate and mini-batch size as
[Equation]
Let N K be the cumulative number of stochastic gradient calls up to step K , i.e. N K = K .
Then we have that
[Equation]
Proof.
Consider the objective improvement in a single step, under one instantiation of the noise.
Use Assumption 2 followed by the deﬁnition of the algorithm.
[Equation]
Take the expectation conditioned on previous iterates, and decompose the mean squared stochastic gradient into its mean and variance.
Technically since we are assuming Hoeffding’s inequality, 4 σis actually an upper bound on the variance.
Note that since 4 σ 2 is the variance bound for each component, the variance bound for the full vector will be 4 dσ 2 .
[Equation]
Plugging in the learning rate schedule, and using that 1 k +1 ≤ 1 √ k +1 , we get that
[Equation]
Sum over k and we get that
[Equation]
And rearranging yields the result.