L APLACIAN S MOOTHING G RADIENT D ESCENT
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a class of very simple modiﬁcations of gradient descent and stochastic gradient descent.
We show that when applied to a large variety of machine learning problems, ranging from softmax regression to deep neural nets, the proposed surrogates can dramatically reduce the variance and improve the generalization accuracy.
The methods only involve multiplying the usual (stochastic) gradient by the inverse of a positive deﬁnitive matrix coming from the discrete Laplacian or its high order generalizations.
The theory of Hamilton-Jacobi partial differential equations demonstrates that the implicit version of new algorithm is almost the same as doing gradient descent on a new function which (i) has the same global minima as the original function and (ii) is “more convex”.
We show that optimization algorithms with these surrogates converge uniformly in the discrete Sobolev H pσ sense and reduce the optimality gap for convex optimization problems.
We implement our algorithm into both PyTorch and Tensorﬂow platforms which only involves changing of a few lines of code.
The code will be available on Github.
I NTRODUCTION
Stochastic gradient descent (SGD) has been the workhorse for solving large-scale machine learning problems (Bottou et al., 2018).
It gives rise to a family of algorithms that make training of deep neural nets (DNN) practical, which is believed to somehow implicitly smooth the loss function of the DNN (Jastrzebski et al., 2018).
Many efforts have been carried out to improve training and generalization of DNN by directly searching for ﬂat minima (Keskar et al., 2017; Chaudhari et al., 2017; 2016).
An alternative view of SGD’s magic comes from the theory of uniform stability (Bousquet &
Elisseeff, 2002; Duchi et al., 2011; Hardt et al., 2016; Bottou et al., 2016; Gonen & Shalev-Shwartz, 2017).
The noise in SGD, on the one hand, helps gradient-based optimization algorithms circumvent spurious local minima and reach those that generalize well (Schmidhuber, 2014).
On the other hand, it slows down the convergence of regular gradient descent (GD).
To recover the linear convergence rate for strongly convex functions, several interesting variance reduction algorithms have been proposed, e.g., SAGA (Defazio & Bach, 2014) and SVRG (Johoson & Zhang, 2013).
These algorithms have a certain amount of difﬁculty in training DNN.
SAGA has a relatively high space complexity in storing the gradient for many samples. SVRG requires computation of the full batch gradient.
In this work, we propose a carefully designed positive deﬁnite matrix to smooth and to reduce variance of the (stochastic) gradient on-the-ﬂy.
The resulting surrogate tends to reduce noise in SGD and improve training of DNN.
We call this procedure Laplacian smoothing.
The gradient smoothing can be done by multiplying the gradient by the inverse of the following circulant convolution matrix
[Equation]
for some positive constant σ ≥ 0 .
In fact, we can write A σ =
I − σ L , where I is the identity matrix, and L is the discrete one-dimensional Laplacian which acts on indices.
We deﬁne the (periodic) forward ﬁnite difference matrix as
D + =   −  . . .
− . . . −. . .
−  .
Then, we have A σ =
I − σ D − D + , where D − = − D (cid:62) + is the backward ﬁnite difference.
The resulting Laplacian smoothing stochastic gradient descent (LS-SGD) requires negligible extra computational cost and generalizes better than the standard SGD.
When the Hessian has a poor condition number, gradient descent performs poorly.
In this case, the derivative increases rapidly in one direction, while increasing slowly in others.
Gradient smoothing can avoid jitter along steep directions and help make progress in shallow directions (Li & et al, 2018).
Moreover, we show that the operator A − 1 σ plays role as a denoiser which enables better convergence in the presence of a very noisy stochastic gradient.
The implicit version of our proposed approach is linked to an unusual HamiltonJacobi partial differential equation (HJ-PDE) whose solution makes the original loss function more convex while retaining its ﬂat (and global) minima, and essentially works on this surrogate function with a much better landscape.
H AMILTON -J ACOBI PDE S AND C ONVEXIFICATION
Machine learning problems are generally formulated as ﬁnding the optimal parameters w of a parametric function y
=
h
( x , w ) , such that for an input x , the output y is close to the ground-truth.
The optimal w can be obtained by minimizing an empirical risk function, f
( X, Y, w ) .
= f ( w ) , given the training data { X, Y } .
We start from the following unusual HJ-PDE with f ( w ) as initial condition
[Equation]
By the Hopf-Lax formula (Evans, 2010), the unique viscosity solution to Eq. (2) is represented by
[Equation]
This viscosity solution u ( w , t ) makes f ( w ) ”more convex”, an intuitive deﬁnition and theoretical explanation of ”more convex” can be found in (Chaudhari et al., 2017; 2016), by bringing down the local maxima while retaining and widening local minima.
An illustration of this is shown in Fig.
If we perform the smoothing GD with proper step size on the function u ( w , t ) , it is easier to reach the global or at least a ﬂat minima of the original nonconvex function f ( w ) .
Proposition 1.
Suppose f ( w ) is differentiable, the LS-GD on u ( w , t )
[Equation]
is equivalent to the smoothing implicit GD on f ( w )
[Equation]
All the proofs here and below are provided in the appendix.
2. L APLACIAN S MOOTHING G RADIENT D ESCENT Laplacian smoothing
implicit gradient descent requires inner iterations as used in (Chaudhari et al., 2017), which is computationally expensive.
We consider the following explicit scheme
[Equation]
Intuitively, compared to the standard GD, this scheme smooths the gradient on-the-ﬂy by an elliptic smoothing operator.
We adopt fast Fourier transform (FFT) to compute A − 1 σ ∇ f ( w k ) , which is available in both PyTorch (Paszke et al., 2017) and TensorFlow (Abadi et al., 2016).
Given a vector g , a smoothed vector d can be obtained by computing d =
A − 1 σ g .
This is equivalent to g =
d − σ v ∗ d , where v = [ − 2 , 1 , 0 , · · · , 0 , 1] (cid:62) and ∗ is the convolution operator.
Therefore
[Equation]
where we use component-wise division, ﬀt and iﬀt are the FFT and inverse FFT, respectively.
Hence, the gradient smoothing can be done in quasilinear time.
This additional time complexity is almost the same as performing a one step update on the weights vector w .
For many machine learning models, we may need to concatenate the parameters into a vector.
This reshape might lead to some ambiguity, nevertheless, based on our tests, both row and column majored reshaping work for the LS-GD algorithm.
Moreover, in deep learning cases, the weights in different layers might have different physical meanings.
We then perform layer-wise gradient smoothing, instead.
Remark 1.
In image processing, the Sobolev gradient (Jung et al., 2009) involves a multidimensional Laplacian operator which operates on w , is different from the one-dimensional discrete
Laplacian operator employed in our LS-GD scheme, which operates on indices.
We ﬁrst show that LS-GD can help bypass sharp minima and reach the global minima.
We consider the following function, in which we ‘drill’ narrow holes on a smooth convex function,
[Equation]
where the summation is taken over the index set { i ∈ N | 0 ≤ i < 4 π } , r and β are the parameters that determine the location and narrowness of the local minima and are set to 1 and 1 √ 500 , respectively.
We do GD and LS-GD starting from a random point in the neighborhoods of the narrow minima, i.e., ( x 0 , y 0 , z 0 ) ∈ { (cid:83)
i U δ ( r sin( i 2 ) + π, r cos( i 2 ) + π ) | 0 ≤ i < 4 π, i ∈ N } , where U δ ( P ) is a neighborhood of the point P with radius δ .
Our experiments (Fig. 2) show that, if δ ≤ 0 . 2 , GD will converge to narrow local minima, while LS-GD convergences to wider global minima.
[Figure]
2. G ENERALIZED S MOOTHING G RADIENT D ESCENT
We can generalize A σ to the n th order discrete hyper-diffusion operator as follows
[Equation]
Each row of the discrete Laplacian operator L consists of an appropriate arrangement of weights in central ﬁnite difference approximation to the 2nd order derivative.
Similarly, each row of L n is an arrangement of the weights of the central ﬁnite difference to approximate the 2 n th order derivative.
Remark 2.
The n th order smoothing operator I + ( − 1) n σ L n can only be applied to the problem with dimension at least 2 n + 1 .
Otherwise, we need to add dummy variables to the object function.
Again, we apply FFT to compute the smoothed gradient vector.
For a given gradient vector g , the smoothed surrogate, ( A nσ ) − 1 g .
= d , can be obtained by solving g = d
+ ( − 1) n σ v n ∗ d , where v n = ( c nn +1 , c nn +2 , · · · , c n 2 n +1 , 0 , · · · , 0 , c n 1 , c n 2 , · · · , c nn − 1 , c nn ) is a vector of the same dimension as the gradient to be smoothed.
And the coefﬁcient vector c n = ( c n 1 , c n 2 , · · · , c n 2 n +1 ) can be obtained recursively by the following formula
[Equation]
Remark 3.
The computational complexities for different order smoothing schemes are the same when the FFT is utilized for computing the surrogate gradient.
R EDUCE O PTIMALITY G AP IN SGD
We show advantages of the LS-(S)GD and generalized schemes for convex optimization.
Consider ﬁnding the minima x ∗ of the quadratic function f ( x ) deﬁned in Eq. (5) by different schemes.
[Equation]
To simulate SGD, we add Gaussian noise to the gradient vector, i.e., at a given point x , we have
[Equation]
where the scalar (cid:15) controls the noise level, N ( 0 , I ) is the vector with zero mean and unit variance in each coordinate.
The corresponding numerical schemes can be formulated as
[Equation]
where σ is the smoothing parameter selected to be 10 . 0 to kill the intense noise.
We take diminishing step sizes with initial values 0 .
1 for SGD/smoothed SGD; 0 . 9 and 1 . 8 for GD/smoothed GD, respectively.
Without noise, the smoothing allows us to take larger step sizes, rounding to the ﬁrst digit, 0 . 9 and 1 .
9 are the largest suitable step size for GD and smoothed version here.
We compare constant learning rate and exponentially decaying learning rate, i.e., after every 1000 iteration, the learning rate is divided by 10.
We apply different schemes that corresponding to n = 0 , 1 , 2 in Eq. (6) to the problem Eq. (5), with the initial point x 0 = (1 , 1 , · · · , 1) .
Figure.
3 shows the iteration v.s. optimality gap when the constant learning rate is applied to different noise levels.
In the noise free case, all three schemes converge linearly, but gradient smoothing has a smaller decay constant due to its increased condition number.
When there is noise, our smoothed gradient helps to reduce the optimality gap and converges faster after a few iterations.
[Equation]
The exponentially decaying learning rate helps our smoothed SGD to reach a point with a smaller optimality gap, and the higher order smoothing further reduce the optimality gap, as shown in Fig.
One simple reason for this in the noisy case is because of the noise removal properties of the smoothing operators.
The inﬂuence of the learning rate is still under investigation.
We establish the convergence of our proposed smoothing gradient descent algorithms.
3. S OME P ROPERTIES OF L APLACIAN S MOOTHING G RADIENT D ESCENT
We say the objective function f has L -Lipschitz gradient, if for any w , u ∈ R m , we have (cid:107)∇ f ( w ) − ∇ f ( u ) (cid:107) ≤ L (cid:107) w − u (cid:107) , and f is a -strongly convex, if (cid:104)∇ f ( w ) − ∇ f ( u ) , w − u (cid:105) ≥ a (cid:107) w − u (cid:107) 2 .
We deﬁne the vector norm induced by any matrix A as (cid:107) w (cid:107) A := (cid:112) (cid:104) w , Aw (cid:105) .
[Figure]
Proposition 2.
Suppose f is convex with the global minimizer w ∗ , and f ∗ =
f ( w ∗ ) .
Consider the following iteration with constant learning rate η >
[Equation]
where g k is the sampled gradient in the k th iteration at w k satisfying E [ g k ] = ∇ f ( w k ) .
Denote G A nσ := lim K →∞ 1 K (cid:80) K − 1 k =0 (cid:107) g k (cid:107)
( A nσ ) − 1 and w K := (cid:80) K − 1 k =0 w k /K
the ergodic average of iterates.
Then the optimality gap is
[Equation]
Note that (cid:107) g (cid:107) ( A nσ ) − 1 generally decreases in n unless g is constant, which indicates that a bigger n implies smaller optimality gap.
This is consistent with the experimental results above.
Proposition 3.
Suppose f is L -Lipschitz smooth and a -strongly convex with the global minimizer w ∗ .
Consider the generalized smoothing gradient descent algorithm
[Equation]
where g k is the sampled gradient in the k th iteration at w k satisfying E (cid:2) g k (cid:3)
= ∇ f ( w k ) and
E (cid:104) (cid:107) g k (cid:107)
( A nσ ) − 1 (cid:105) ≤ C 0 + C 1 (cid:107)∇ f ( w k ) (cid:107) 2 for all k ∈ N .
If we take η k = Ck +1 for some C > 0 , then we have
[Equation]
[Equation]
i.e., we have H nσ uniform convergence in σ of { w k } in expectation.
The H nσ norm of w is deﬁned by (cid:107) w (cid:107) nσ := (cid:107) w (cid:107)
A
[Equation]
= (cid:112) (cid:104) w , A nσ w (cid:105) .
Proposition 4. Consider the algorithm w k +1 =
w k − η k ( A n σ ) − 1 ∇ f ( w k ) .
Suppose f is convex and L -Lipschitz smooth.
If the step size satisﬁes 0 < ¯ η ≤ η ≤ ¯ η < 2 L .
Then lim t →∞ (cid:107)∇ f ( w k ) (cid:107) → 0 .
Moreover, if the Hessian ∇ 2 f of f is continuous with w ∗ being the global minimizer of f , and ¯ η (cid:107)∇ 2 f (cid:107) < 1 , then (cid:107) w k − w ∗ (cid:107)
A nσ → 0 as k → ∞ , and the convergence is linear and independent of σ .
In what follows, we present the noise reduction properties of the proposed smoothing operator
A − 1 σ .
Proposition 5.
For any vector g ∈ R m , d = A − 1 σ g , let j max = arg max i
d i and j min = arg min i d i .
We have max i d i =
d j max ≤ g j max ≤ max
i g i and min i d i =
d j min ≥ g j min ≥ min i g i .
Proposition 6.
The operator A − 1 σ preserves the sum of components.
For any g ∈ R m and d = A − 1 σ g , we have (cid:80) j d j = (cid:80) j g j , or equivalently, 1 (cid:62) d = 1 (cid:62) g .
Proposition 7.
Given any vector g ∈ R m and d = A − 1 σ g , then
[Equation]
The above inequality is strict unless g = d is a constant vector.
In particular, we have (cid:107) d (cid:107) ≤ (cid:107) g (cid:107) and (cid:107) D + d (cid:107) ≤ 1 √ σ (cid:107) g (cid:107) .
Let g be the noise vector contained in the stochastic gradient, the above results imply that the extreme values in A − 1 σ g are smaller than those in g (in magnitude), and it also has a much smaller (cid:96) 2 norm.
Proposition 8.
For any g ∈ R m , deﬁne Var( g ) :
= 1 m (cid:107) g (cid:107) 2 − (cid:16) 1 (cid:62) g m (cid:17) 2 be the variance of components in g .
Let d
= A − 1 σ g , then
[Equation]
The inequality is strict unless g = d is a constant vector.
Proposition 8 shows that the component-wise variance of A − 1 σ g is considerably less than that of g , unless g is a constant vector.
Our last result shows that A − 1 σ g has diminishing (cid:96) 1 norm of ﬁnite difference of all orders.
This is an excellent desnoising result.
Proposition 9.
Given vectors g and d = A − 1 σ g , for any p ∈ N , it holds that (cid:107) D p + d (cid:107) 1 ≤ (cid:107) D p + g (cid:107) 1 .
The inequality is strict unless D p + g is a constant vector.
Remark 4.
The above proofs generalize for n > 1 , except for Propositions 5 and 9.
3. S OFTMAX R EGRESSION
Consider applying the proposed optimization schemes to Softmax regression.
We run 200 epochs of SGD and different order smoothing algorithms to maximize the likelihood of Softmax regression with batch size 100.
Based on the results from previous section, we apply the exponentially decay learning rate with initial value 0 .
1 and decay 10 times after every 50 epochs.
We train the model with only 10 % randomly selected MNIST training data and test the trained model on the entire testing images.
We further compare with SVRG under the same setting.
Figure.
5 shows the histograms of generalization accuracy of Softmax regression model trained by SGD ((a)); SVRG ((b)); LSSGD (order 1) ((c)); LS-SGD (oder 2) ((d)).
It is seen that SVRG improves the generalization with higher average accuracy.
But the ﬁrst and second order smoothing schemes signiﬁcantly improve averaged generalization accuracy by more than 1% and reduce the variance over 100 independent trials.
The training loss of these 100 experiments by different optimization algorithms are shown in the appendix.
[Figure]
A PPLICATIONS TO D EEP N EURAL N ETS 4. T RAIN N EURAL N ETS WITH S MALL B ATCH S IZE
Many advanced artiﬁcial intelligence tasks make high demand on training neural nets with extremely small batch size. The milestone technique for this is group normalization (Wu & He, 2018).
In this section, we show that LS-SGD successfully trains DNN with extremely small batch size.
We consider LeNet-5 devised by (LeCun et al., 1998) for MNIST classiﬁcation.
Our network architecture is as follows
[Equation]
The notation conv c,k,m denotes a 2D convolutional layer with c output channels, each of which is the sum of a channel-wise convolution operation on the input using a learnable kernel of size k × k , it further adds ReLU nonlinearity and max pooling with stride size m .
fc 512 is an afﬁne transformation that transforms the input to a vector of dimension 512.
Finally, the tensors are activated by a softmax function.
The MNIST data is ﬁrst passed to the layer input 28 × 28 , and further processed by this hierarchical structure.
We run 100 epochs of both SGD and LS-SGD with initial learning rate 0 .and
divide by 5 after 50 epochs, and use a weight decay of 0 . 0001 and momentum of 0 . 9 .
Figure.
6(a) plots the generalization accuracy on the test set with the LeNet5 trained with different batch sizes.
For each batch size, LS-SGD with σ = 1 . 0 keeps the testing accuracy more than 99 .
4% , SGD reduce the accuracy to 97% when batch size 4 is used.
The classiﬁcation become just a random guess, when the model is trained by SGD with batch size 2.
Small batch size leads to large noise in the gradient, which may make the noisy gradient not along the decent direction, However, Lapacian smoothing rescues this by killing the noise.
4. I MPROVE G ENERALIZATION A CCURACY
The skip connections in ResNet smooth the landscape of the loss function of the classical CNN (He et al., 2016; Li et al., 2017).
This means that ResNet has fewer sharp minima.
On Cifar (Krizhevsky, 2009), we compare the performance of LS-SGD and SGD on ResNet with the preactivated ResNet56 as an illustration.
We take the same training strategy as that used in (He et al., 2016), except that we run 200 epochs with the learning rate decaying by a factor of 5 after everyepochs.
For ResNet, instead of applying LS-SGD for all epochs, we only use LS-SGD in the ﬁrstepochs, and the remaining training is carried out by SGD.
The parameter σ is set to 1 . 0 .
Figure 6(b) depicts one path of the training and generalization accuracy of the neural nets trained by SGD and
LS-SGD, respectively.
It is seen that, even though the training accuracy obtained by SGD is higher than that by LS-SGD, the generalization is however inferior to that of LS-SGD.
We conjecture that this is due to the fact that SGD gets trapped into some sharp but deeper minimum, which ﬁts better than a ﬂat minimum but generalizes worse.
We carry out 25 replicas of this experiments, the histograms of the corresponding accuracy are shown in Fig.
[Figure]
4. T RAINING W ASSERSTERIN GAN
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are notoriously delicate and unstable to train (Arjovsky & Bottou, 2017).
In (M. Arjovsky & Bottou, 2017), Wasserstein-GANs (WGANs) are introduced to combat the instability in the training GANs.
In addition to being more robust in training parameters and network architecture, WGANs provide a reliable estimate of the Earth Mover (EM) metric which correlates well with the quality of the generated samples.
Nonetheless, WGANs training becomes unstable with a large learning rate or when used with a momentum based optimizer (M. Arjovsky & Bottou, 2017).
In this section, we demonstrate that the gradient smoothing technique in this paper alleviates the instability in the training, and improves the quality of generated samples.
Since WGANs with weight clipping are typically trained with RMSProp (Tieleman & Hinton, 2012), we propose replacing the gradient g by a smoothed version g σ =
A − 1 σ g , and also update the running averages using g σ instead of g .
We name this algorithm LS-RMSProp.
To accentuate the instability in training and demonstrate the effects of gradient smoothing, we deliberately use a large learning rate for training the generator.
We compare the regular RMSProp with the LS-RMSProp.
The learning rate for the critic is kept small and trained approximately to convergence so that the critic loss is still an effective approximation to the Wasserstein distance.
To control the number of unknowns in the experiment and make a meaningful comparison using the critic loss, we use the classical RMSProp for the critic, and only apply LS-RMSProp to the generator.
[Figure]
We train the WGANs on the MNIST dataset using the DCGAN (Radford et al., 2015) for both the critic and generator.
In Figure 8 (left), we observe the loss for RMSProp trained with a large learning rate has multiple sharp spikes, indicating instability in the training process.
The samples generated are also lower in quality, containing noisy spots as shown in Figure 9 (a).
In contrast, the curve of training loss for LS-RMSProp is smoother and exhibits fewer spikes.
The generated samples as shown in Fig. 9 (b) are also of better quality and visibly less noisy.
The generated characters shown in Fig. 9 (b) are more realistic compared to the ones shown in Fig.
9 (a). The effects are less pronounced with a small learning rate, but still result in a modest improvement in sample quality as shown in Figure 9 (c) and (d).We also apply LS-RMSProp for training the critic, but do not see a clear improvement in the quality.
This may be because the critic is already trained near optimality during each iteration, and does not beneﬁt much from gradient smoothing.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
4. D EEP R EINFORCEMENT L EARNING
Finally, we apply the LS-SGD to deep reinforcement learning.
We provide a detailed discussion and present the numerical result in the appendix.
C ONCLUDING R EMARKS
Motivated by the theory of Hamilton-Jacobi partial differential equations, we proposed Laplacian smoothing gradient descent and its high order generalizations.
This simple modiﬁcation dramatically reduces the optimality gap in stochastic gradient descent and helps to ﬁnd better minima.
Extensive numerical examples ranging from toy cases to shallow and deep neural nets to generative adversarial networks and to deep reinforcement learning, all demonstrate the advantage of the proposed smoothed gradient.
Several issues remain, in particular devising an on-the-ﬂy adaptive method for choosing the smoothing parameter σ instead of using a ﬁxed value.
A PPENDIX 6. T ECHNICAL P ROOFS
Proposition 1.
Suppose f ( w ) is differentiable, the Laplacian smoothing GD update on u ( w , t )
[Equation]
permits the smoothing implicit gradient descent on f ( w )
[Equation]
Proof of Proposition 1.
We deﬁne
[Equation]
and rewrite u ( w , t ) = inf v z ( w , v , t ) as z ( w , v ( w , t ) , t ) , where v ( w , t )
=
arg min v z ( w , v , t ) .
Then by the Euler-Lagrange equation,
[Equation]
w
[Equation] t [Equation] where J w v ( w , t ) is the Jacobian matrix of v w.r.t.
w .
Notice that ∇ v z ( w , v ( w , t ) , t ) = 0 ,
[Equation]
Letting w = w k and w k +
= v ( w k , t ) =
arg min v z ( w k , v , t ) in the above equalities, we have
[Equation]
In summary, the gradient descent w k +1 = w k − t
A − 1 σ ∇ w u ( w k , t ) is equivalent to the proximal point iteration w k +1 =
arg min v f ( v ) + 12 t (cid:104) v − w k , A σ ( v − w k ) (cid:105) , which yields w k +1 =
w k − t
A − 1 σ ∇ f ( w k +1 ) .
Proposition 2.
Suppose f is convex with the global minimizer w ∗ , and f ∗ =
f ( w ∗ ) .
Consider the following iteration with constant learning rate η >
[Equation]
where g k is the sampled gradient in the k th iteration at w k satisfying E [ g k ] = ∇ f ( w k ) .
Denote G A nσ := lim K →∞ 1 K (cid:80) K − 1 k =0 (cid:107) g k (cid:107)
( A nσ ) − 1 and w K := (cid:80) K − 1 k =0 w k /K
the ergodic average of iterates.
Then the optimality gap is
[Equation]
Proof.
Since f is convex, we have
[Equation]
Furthermore,
[Equation]
where the last inequality is due to (7)
[Equation]
We rearrange the terms and arrive at
[Equation]
Summing over k from 0 to K − 1 and averaging and using the convexity of f , we have
[Equation]
Taking the limit as K → ∞ above establishes the result
[Equation]
Proposition 3.
Suppose f is L -Lipschitz smooth and a -strongly convex.
Consider the generalized smoothing gradient descent algorithm
[Equation]
where g k is the sampled gradient in the k th iteration at w k satisfying E (cid:2) g k (cid:3)
= ∇ f ( w k ) and
E (cid:104) (cid:107) g k (cid:107)
( A nσ ) − 1 (cid:105) ≤ C 0 + C 1 (cid:107)∇ f ( w k ) (cid:107) 2 for all k ∈ N .
If we take η k = Ck +1 for some C > 0 , then we have
[Equation]
[Equation]
i.e., we have H nσ uniform convergence in σ of { w k } in expectation.
The H nσ norm of w is deﬁned by
[Equation]
Proof of Proposition 3.
Since ∇ f ( w ∗ ) = 0 , by strong convexity of f , we have
[Equation]
Moreover, by L -smoothness of f and the fact that (cid:107) A nσ (cid:107) =
1 , we also have
[Equation]
Hence,
[Equation]
where in the ﬁrst inequality we used (cid:107) ( A nσ ) − 1 (cid:107)
=
for all σ and n .
Taking η k = Ck
+ for some proper C > 0 and using induction, one can show that E (cid:104) (cid:107) w k − w ∗
[Equation]
2 A nσ (cid:105) = E
[Equation]
Proposition 4. Consider the algorithm w k +1 =
w k − η k
( A nσ ) − 1 ∇ f ( w k ) .
Suppose f is L -Lipschitz smooth and 0 < ¯ η ≤ η ≤ ¯ η < 2 L .
Then lim t →∞ (cid:107)∇ f ( w k ) (cid:107) → 0 .
Moreover, if the Hessian ∇ 2 f of f is continuous with w ∗ being the minimizer of f , and ¯ η (cid:107)∇ 2 f (cid:107) < 1 , then (cid:107) w k − w ∗ (cid:107)
A nσ → 0 as k → ∞ , and the convergence is linear.
Proof of Proposition 4.
By the Lipschitz continuity of ∇ f and the descent lemma (Bertsekas, 1999), we have
[Equation]
Summing the above inequality over k , we have
[Equation]
Therefore, (cid:107)∇ f ( w k ) (cid:107) 2( A nσ ) − 1 → 0 , and thus (cid:107)∇ f ( w k ) (cid:107) → 0 . For the second claim [Equation]
we have
[Equation]
Therefore,
[Equation]
So if η k (cid:107)∇ 2 f (cid:107) ≤
[Equation]
A
[Equation]
[Equation] = 1 , the result follows.
Proposition 5.
For any vector g ∈ R m , d = A − 1 σ g , let j max = arg max i
d i and j min = arg min i d i .
We have max i d i =
d j max ≤ g j max ≤ max
i g i and min i d i =
d j min ≥ g j min ≥ min i g i .
Proof of Proposition 5.
Since g =
A σ d , it holds that
[Equation]
where periodicity of subindex are used if necessary.
Since 2 d j max − d j max − 1 − d j max +1 ≥ 0 , We have max i d i =
d j max ≤ g j max ≤ max
i g i .
A similar argument can show that min i d i =
d j min ≥ g j min ≥ min i g i .
Proposition 6.
The operator A − 1 σ preserves the sum of components.
For any g ∈ R m and d = A − 1 σ g , we have (cid:80) j d j = (cid:80) j g j , or equivalently, 1 (cid:62) d = 1 (cid:62) g .
Proof of Proposition 6.
Since g = A σ d ,
[Equation]
where we used D + 1 = 0 .
Proposition 7.
Given any vector g ∈ R m and d = A − 1 σ g , then
[Equation]
The above inequality is strict unless g = d is a constant vector.
In particular, we have (cid:107) d (cid:107) ≤ (cid:107) g (cid:107) and (cid:107) D + d (cid:107) ≤ 1 √ σ (cid:107) g (cid:107) .
Proof of Proposition 7.
By the deﬁnition of A σ ,
[Equation]
[Equation]
Therefore, pre-multiplying by d (cid:62) on both sides, we have
[Equation]
In particular, (cid:107) d (cid:107) ≤ (cid:107) g (cid:107) and σ (cid:107) D + d (cid:107) 2 ≤ (cid:107) d (cid:107)(cid:107) g (cid:107) ≤ (cid:107) g (cid:107) 2 , so (cid:107) D + d (cid:107) ≤ 1 √ σ (cid:107) g
(cid:107) .
All the inequalities are strict unless (cid:107) D + d (cid:107) = 0 , and g =
d is a constant vector.
Proposition 8.
For any g ∈ R m , deﬁne Var( g ) :
= 1 m (cid:107) g (cid:107) 2 − (cid:16) 1 (cid:62) g m (cid:17) 2 be the variance of components in g .
Let d
= A − 1 σ g , then
[Equation]
The inequality is strict unless g = d is a constant vector.
Proof of Proposition 8.
Since 1 (cid:62) g = 1 (cid:62) d and (cid:107) d (cid:107) + σ (cid:107) D + d (cid:107) 2 (cid:107) d (cid:107)
[Equation]
The inequality is strict unless (cid:107) D + d (cid:107) = 0 , and g =
d is a constant vector.
Proposition 9.
Given vectors g and d = A − 1 σ g , for any p ∈ N , it holds that (cid:107) D p + d (cid:107) 1 ≤ (cid:107) D p + g (cid:107) 1 .
The inequality is strict unless D p + g is a constant vector.
Proof of Proposition 9.
Since (1 + 2 σ ) d i = g i + σd i +
+ σd i − 1 , for any p ∈ N , we have
[Equation]
D
[Equation]
+
[Equation]
−
[Equation]
So
[Equation]
The inequality is strict if there are sign changes among the ( D p + d ) i − 1 , ( D p + d ) i , ( D p + d ) i +1 .
Summing over i and using periodicity, we have
[Equation]
and the result follows.
The inequality is strict unless D p + g is a constant vector.
6. I TERATION V . S . L OSS FOR S OFTMAX R EGRESSION
In this part, we show the training loss evolution in training Softmax regression model, respectively, by SGD, SVRG, LSGD with ﬁrst and second order smoothing.
As illustrated in Fig.
10, all the optimization algorithms reduce loss of the model on the training set.
At each iteration, amongindependent experiments, SGD has the largest variance, SGD with ﬁrst order smoothed gradient signiﬁcantly reduces the variance of loss function.
The second order smoothing can further reduce variance of loss.
The variance of loss in each iteration among 100 experiments is minimized when
SVRG is use to train the Softmax model.
However, the generalization performance of the model trained by SVRG is not as good as the ones trained by LS-SGD or higher order smoothed gradient descent.
6. D EEP R EINFORCEMENT L EARNING
Deep reinforcement learning (DRL) has been applied to playing games including Cartpole (Brockman et al., 2016), Atari (Mnih et al., 2013), Go (Silver & et al, 2016; Mnih & et al, 2015).
DNN plays a vital role in approximating the Q-function or policy function.
We apply the Laplacian smoothed gradient to train the policy function to play the Cartpole game.
We apply the standard procedure to train the policy function by using the policy gradient (Brockman et al., 2016).
We use the following network to approximate the policy function:
[Equation]
[Figure]
The network is trained by RMSProp and LS-RMSProp with σ = 1 . 0 , respectively.
The learning rate and other related parameters are set to be the default ones in PyTorch.
The training is stopped once the average duration of 5 consecutive episodes is more than 490.
In each training episode, we set the maximal steps to be 500.
Left and right panels of Fig.
11 depict a training procedure by using RMSProp and LS-RMSProp, respectively.
We see that Laplacian smoothed gradient takes fewer episodes to reach the stopping criterion.
Moreover, we run the above experiments 5 times independently, and apply the trained model to play Cartpole.
The game lasts more than 1000 steps for all the 5 models trained by LS-RMSProp, while only 3 of them lasts more than 1000 steps when the model is trained by vanilla RMSProp.
[Figure]