B LOCK -N ORMALIZED G RADIENT M ETHOD : A N E MPIRICAL S TUDY FOR T RAINING D EEP N EURAL N ETWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose a generic and simple strategy for utilizing stochastic gradient information in optimization.
The technique essentially contains two consecutive steps in each iteration: 1) computing and normalizing each block (layer) of the mini-batch stochastic gradient; 2) selecting appropriate step size to update the decision variable (parameter) towards the negative of the block-normalized gradient.
We conduct extensive empirical studies on various non-convex neural network optimization problems, including multi layer perceptron, convolution neural networks and recurrent neural networks.
The results indicate the blocknormalized gradient can help accelerate the training of neural networks.
In particular, we observe that the normalized gradient methods having constant step size with occasionally decay, such as SGD with momentum, have better performance in the deep convolution neural networks, while those with adaptive step sizes, such as Adam, perform better in recurrent neural networks.
Besides, we also observe this line of methods can lead to solutions with better generalization properties, which is conﬁrmed by the performance improvement over strong baselines.
I NTRODUCTION
Continuous optimization is a core technique for training non-convex sophisticated machine learning models such as deep neural networks (Bengio, 2009).
Compared to convex optimization where a global optimal solution is expected, non-convex optimization usually aims to ﬁnd a stationary point or a local optimal solution of an objective function by iterative algorithms.
Among a large volume of optimization algorithms, ﬁrst-order methods, which only iterate with the gradient information of objective functions, are widely used due to its relatively low requirement on memory space and computation time, compared to higher order algorithms.
In many machine learning scenarios with large amount of data, the full gradient is still expensive to obtain, and hence the unbiased stochastic version will be adopted, as it is even more computationally efﬁcient.
In this paper, we are particularly interested in solving the deep neural network training problems with stochastic ﬁrst order methods.
Compared to other non-convex problems, deep neural network training additionally has the following challenge: gradient may be vanishing and/or exploding.
More speciﬁcally, due to the chain rule (a.k.a. backpropagation), the original gradient in the low layers will become very small or very large because of the multiplicative effect of the gradients from the upper layers, which is usually all smaller or larger than 1.
As the number of layers in the neural network increases, the phenomenon of vanishing or exploding gradients becomes more severe such that the iterative solution will converge slowly or diverge quickly.
We aim to alleviate this problem by block-wise stochastic gradient normalization , which is constructed via dividing the stochastic gradient by its norm.
Here, each block essentially contains the variables of one layer in a neural network, so it can also be interpreted as layer-wise gradient normalization.
Compared to the regular gradient, normalized gradient only provides an updating direction but does not incorporate the local steepness of the objective through its magnitude, which helps to control the change of the solution through a well-designed step length.
Intuitively, as it constrains the magnitude of the gradient to be 1, it should to some extent prevent the gradient vanishing or exploding phenomenon.
In fact, as showed in (Hazan et al., 2015; Levy, 2016), normalized gra- dient descent (NGD) methods are more numerically stable and have better theoretical convergence properties than the regular gradient descent method in non-convex optimization.
Once the updating direction is determined, step size (learning rate) is the next important component in the design of ﬁrst-order methods.
While for convex problems there are some well studied strategies to ﬁnd a stepsize ensuring convergence, for non-convex optimization, the choice of step size is more difﬁcult and critical as it may either enlarge or reduce the impact of the aforementioned vanishing or exploding gradients.
Among different choices of step sizes, the constant or adaptive feature-dependent step sizes are widely adopted. On one hand, stochastic gradient descent (SGD) + momentum + constant step size has become the standard choice for training feed-forward networks such as Convolution Neural Networks (CNN). Ad-hoc strategies like decreasing the step size when the validation curve plateaus are well adopted to further improve the generalization quality.
On the other hand, different from the standard step-size rule which multiplies a same number to each coordinate of gradient, the adaptive feature-dependent step-size rule multiplies different numbers to coordinates of gradient so that different parameters in the learning model can be updated in different paces.
For example, the adaptive step size invented by (Duchi et al., 2011) is constructed by aggregating each coordinates of historical gradients.
As discussed by (Duchi et al., 2011), this method can dynamically incorporate the frequency of features in the step size so that frequently occurring coordinates will have a small step sizes while infrequent features have long ones.
The similar adaptive step size is proposed in (Kingma & Ba, 2014) but the historical gradients are integrated into feature-dependent step size by a different weighting scheme.
In this paper, we propose a generic framework using the mini-batch stochastic normalized gradient as the updating direction (like (Hazan et al., 2015; Levy, 2016)) and the step size is either constant or adaptive to each coordinate as in (Duchi et al., 2011; Kingma & Ba, 2014).
Our framework starts with computing regular mini-batch stochastic gradient, which is immediately normalized layer-wisely.
The normalized version is then plugged in the constant stepsize with occasional decay, such as SGD+momentum, or the adaptive step size methods, such as Adam (Kingma & Ba, 2014) and AdaGrad (Duchi et al., 2011).
The numerical results shows that normalized gradient always helps to improve the performance of the original methods especially when the network structure is deep.
It seems to be the ﬁrst thorough empirical study on various types of neural networks with this normalized gradient idea.
Besides, although we focus our empirical studies on deep learning where the objective is highly non-convex, we also provide a convergence proof under this framework when the problem is convex and the stepsize is adaptive in the appendix.
This convergence under the non-convex case will be a very interesting and important future work.
The rest of the paper is organized as follows.
In Section 2, we brieﬂy go through the previous work that are related to ours.
In Section 3, we formalize the problem to solve and propose the generic algorithm framework.
In Section 4, we conduct comprehensive experimental studies to compare the performance of different algorithms on various neural network structures.
We conclude the paper in Section 5.
Finally, in the appendix, we provide a concrete example of this type of algorithm and show its convergence property under the convex setting.
R ELATED W ORK
A pioneering work on normalized gradient descent (NGD) method was by Nesterov (Nesterov, 1984) where it was shown that NGD can ﬁnd a (cid:15) -optimal solution within O ( 1 (cid:15) 2 ) iterations when the objective function is quasi-convex.
Kiwiel (Kiwiel, 2001) and Hazan et al (Hazan et al., 2015) extended NGD for upper semi-continuous quasi-convex objective functions and local-quasi-convex objective functions, respectively, and achieved the same iteration complexity.
Moreover, Hazan et al (Hazan et al., 2015) showed that NGD’s iteration complexity can be reduced to O ( 1 (cid:15) ) if the objective function is local-quasi-convex and locally-smooth.
A stochastic NGD algorithm is also proposed by Hazan et al (Hazan et al., 2015) which, if a mini-batch is used to construct the stochastic normalized gradient in each iteration, ﬁnds (cid:15) -optimal solution with a high probability for locallyquasi-convex functions within O ( 1 (cid:15) 2 ) iterations.
Levy (Levy, 2016) proposed a Saddle-Normalized
Gradient Descent (Saddle-NGD) method, which adds a zero-mean Gaussian random noise to the stochastic normalized gradient periodically.
When applied to strict-saddle functions with some ad- minimum
Analogous yet orthogonal to the gradient normalization ideas have been proposed for the deep neural network training.
For example, batch normalization (Ioffe & Szegedy, 2015) is used to address the internal covariate shift phenomenon in the during deep learning training.
It beneﬁts from making normalization a part of the model architecture and performing the normalization for each training mini-batch.
Weight normalization (Salimans & Kingma, 2016), on the other hand, aims at a reparameterization of the weight vectors that decouples the length of those weight vectors from their direction.
Recently (Neyshabur et al., 2015) proposes to use path normalization, an approximate path-regularized steepest descent with respect to a path-wise regularizer related to max-norm regularization to achieve better convergence than vanilla SGD and AdaGrad.
Perhaps the most related idea to ours is Gradient clipping.
It is proposed in (Pascanu et al., 2013) to avoid the gradient explosion, by pulling the magnitude of a large gradient to a certain level.
However, this method does not do anything when the magnitude of the gradient is small.
Adaptive step size has been studied for years in the optimization community.
The most celebrated method is the line search scheme.
However, while the exact line search is usually computational infeasible, the inexact line search also involves a lot of full gradient evaluation.
Hence, they are not suitable for the deep learning setting.
Recently, algorithms with adaptive step sizes start to be applied to the non-convex neural network training, such as AdaGrad (Duchi et al., 2012), Adam (Kingma & Ba, 2014) and RMSProp (Hinton et al.).
However they directly use the unnormalized gradient, which is different from our framework.
Singh et al.
(2015) recently proposes to apply layer-wise speciﬁc step sizes, which differs from ours in that it essentially adds a term to the gradient rather than normalizing it. Recently Wilson et al. (2017) ﬁnds the methods with adaptive step size might converge to a solution with worse generalization. However, this is orthogonal to our focus in this paper.
A LGORITHM F RAMEWORK
In this section, we present our algorithm framework to solve the following general problem:
[Equation]
where x = ( x 1 , x 2 , . . .
x B ) ∈ R d with x i ∈ R
d i and (cid:80) Bi
=1 d i = d , ξ is a random variable following some distribution P , F ( · , ξ ) is a loss function for each ξ and the expectation E is taken over ξ .
Now our goal is to minimize the objective function f over x , where x can be the parameters of a machine learning model when (1) corresponds to a training problem.
Here, the parameters are partitioned into B blocks.
The problem of training a neural network is an important instance of (1), where each block of parameters x i can be viewed as the parameters associated to the i th layer in the network.
We propose the generic optimization framework in Algorithm 1.
In iteration t , it ﬁrstly computes the partial (sub)gradient F (cid:48) i ( x t , ξ t ) of F with respect to x i for i = 1 , 2 , . . .
, at x = x t with a mini-batch data ξ t , and then normalizes it to get a partial direction g it =
[Equation]
We deﬁne g t =
( g 1 t , g 2 t , . .
, g Bt ) .
The next is to ﬁnd d adaptive step sizes τ t ∈ R d with each coordinate of τ t corresponding to a coordinate of x .
We also partition τ t in the same way as x so that τ t = ( τ 1 t , τ 2 t , . . .
, τ Bt ) ∈ R B with τ it ∈ R d i .
We use τ t as step sizes to update x t to x t +1 as x t +
= x
t − τ t ◦ g t , where ◦ represents coordinate-wise (Hadamard) product.
In fact, our framework can be customized to most of existing ﬁrst order methods with ﬁxed or adaptive step sizes, such as SGD,
AdaGrad(Duchi et al., 2011), RMSProp (Hinton et al.) and Adam(Kingma & Ba, 2014), by adopting their step size rules respectively.
N UMERICAL E XPERIMENTS
Basic Experiment Setup In this section, we conduct comprehensive numerical experiments on different types of neural networks.
The algorithms we are testing are SGD with Momentum (SGDM), AdaGrad (Duchi et al., 2013), Adam (Kingma & Ba, 2014) and their block-normalized gradient counterparts, which are denoted with sufﬁx “NG”.
Speciﬁcally, we partition the parameters
1: Choose x 1 ∈ R d .
2: for t = 1 , 2 , ..., do 3:
Sample a mini-batch of data ξ t and compute the partial stochastic gradient g it =
F (cid:48) i ( x t ,ξ t ) (cid:107) F (cid:48)
i ( x t ,ξ t ) (cid:107)4:
Let g t = ( g 1 t , g 2 t , . .
, g Bt ) and choose step sizes τ t ∈ R d .
5: x t +1 =
x
t − τ t ◦ g t 6: end for into block as x = ( x 1 , x 2 , . . .
, x B ) such that x i corresponds to the vector of parameters (including the weight matrix and the bias/intercept coefﬁcients) used in the i th layer in the network.
Our experiments are on four diverse tasks, ranging from image classiﬁcation to natural language processing.
The neural network structures under investigation include multi layer perceptron, longshort term memory and convolution neural networks.
To exclude the potential effect that might be introduced by advanced techniques, in all the experiments, we only adopt the basic version of the neural networks, unless otherwise stated.
The loss functions for classiﬁcations are cross entropy, while the one for language modeling is log perplexity.
Since the computational time is proportional to the epochs, we only show the performance versus epochs.
Those with running time are similar so we omit them for brevity.
For all the algorithms, we use their default settings.
More speciﬁcally, for Adam/AdamNG, the initial step size scale α = 0 .
001 , ﬁrst order momentum β 1 = 0 .
9 , second order momentum β 2 = 0 .
999 , the parameter to avoid division of zero (cid:15) = 1 e − 8 ; for AdaGrad/AdaGradNG, the initial step size scale is 0.01.
4. M ULTI L AYER P ERCEPTRON FOR MNIST I MAGE C LASSIFICATION
The ﬁrst network structure we are going to test upon is the Multi Layer Perceptron (MLP).
We will adopt the handwritten digit recognition data set MNIST 1 Lecun et al.
(1998), in which, each data is an image of hand written digits from { 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 0 } .
There are 60k training and 10k testing examples and the task is to tell the right number contained in the test image.
Our approach is applying MLP to learn an end-to-end classiﬁer, where the input is
the raw 28 × 28 images and the output is the label probability.
The predicted label is the one with the largest probability.
In each middle layer of the MLP, the hidden unit number are 100, and the ﬁrst and last layer respectively contain 784 and 10 units.
The activation functions between layers are all sigmoid and the batch size is 100 for all the algorithms.
We choose different numbers of layer from { 6 , 12 , 18 } .
The results are shown in Figure 1.
Each column of the ﬁgures corresponds to the training and testing objective curves of the MLP with a given layer number.
From left to right, the layer numbers are respectively 6, 12 and 18.
We can see that, when the network is as shallow as containing 6 layers, the normalized stochastic gradient descent can outperform its unnormalized counterpart, while the Adam and AdaGrad are on par with or even slightly better than their unnormalized versions.
As the networks become deeper, the acceleration brought by the gradient normalization turns more signiﬁcant.
For example, starting from the second column, AdamNG outperforms Adam in terms of both training and testing convergence.
In fact, when the network depth is 18, the AdamNG can still converge to a small objective value while Adam gets stuck from the very beginning.
We can observe the similar trend in the comparison between AdaGrad (resp.
SGDM) and AdaGradNG (resp.
SGDMNG).
On the other hand, the algorithms with adaptive step sizes can usually generate a stable learning curve.
For example, we can see from the last two column that SGDNG causes signiﬁcant ﬂuctuation in both training and testing curves.
Finally, under any setting, AdamNG is always the best algorithm in terms of convergence performance.
[Figure]
4. R ESIDUAL N ETWORK ON CIFAR10 AND CIFAR
Datasets In this section, we benchmark the methods on CIFAR (both CIFAR10 and CIFAR100) datasets with the residual networks He et al.
(2016a), which consist mainly of convolution layers and each layer comes with batch normalization (Ioffe & Szegedy, 2015).
CIFAR10 consists of 50,000 training images and 10,000 test images from 10 classes, while CIFAR100 from 100 classes.
Each input image consists of 32 × 32 pixels.
The dataset is preprocessed as described in He et al.
(2016a) by subtracting the means and dividing the variance for each channel.
We follow the same data augmentation in He et al. (2016a) that 4 pixels are padded on each side, and a 32 × 32 crop is randomly sampled from the padded image or its horizontal ﬂip.
Algorithms We adopt two types of optimization frameworks, namely SGD and Adam 2 , which respectively represent the constant step size and adaptive step size methods.
We compare the performances of their original version and the layer-normalized gradient counterpart. We also investigate how the performance changes if the normalization is relaxed to not be strictly 1.
In particular, we ﬁnd that if the normalized gradient is scaled by its variable norm with a ratio, which we call NG adap and deﬁned as follows, [Equation] we can get lower testing error. The subscript “adap” is short for “adaptive”, as the resulting norm of the gradient is adaptive to its variable norm, while α is the constant ratio.
Finally, we also compare with the gradient clipping trick that rescales the gradient norm to a certain value if it is larger than that threshold. Those methods are with sufﬁx “CLIP”. Parameters In the following, whenever we need to tune the parameter, we search the space with a holdout validation set containing 5000 examples.
For SGD+Momentum method, we follow exactly the same experimental protocol as described in He et al. (2016a)
and adopt the publicly available Torch implementation 3 for residual network.
In particular, SGD is used with momentum of 0.9, weight decay of 0.0001 and mini-batch size of 128.
The initial learning rate is 0.1 and dropped by a factor of 0.1 at 80, 120 with a total training ofepochs. The weight initialization is the same as He et al. For Adam, we search the initial learning rate in range { 0 . 0005 , 0 . 001 , 0 . 01 } with the base algorithm Adam.
We then use the best learning rate, i.e., 0.001, for all the related methods AdamCLIP, AdamNG, and AdamNG adap .
Other setups are the same as the SGD. In particular, we also adopt the manually learning rate decay here, since, otherwise, the performance will be much worse. We adopt the residual network architectures with depths L = { 20 , 32 , 44 , 56 , 110 } on both CIFAR10 and CIFAR100.
For the extra hyper-parameters, i.e., threshold of gradient clipping and scale ratio of NG adap , i.e., α , we choose the best hyper-parameter from the 56-layer residual network.
In particular, for clipping, the searched values are { 0 . 05 , 0 . 1 , 0 . 5 , 1 , 5 } , and the best value is 0.1.
For the ratio α , the searched values are { 0 . 01 , 0 . 02 , 0 . 05 } and the best is 0.02.
Results For each network structure, we make 5 runs with random initialization. We report the training and testing curves on CIFAR10 and CIFAR100 datasets with deepest network Res-110 in Figure 2.
We can see that the normalized gradient methods (with sufﬁx “NG”) converge the fastest in training, compared to the non-normalized counterparts.
While the adaptive version NG adap is not as fast as NG in training, however, it can always converge to a solution with lower testing error, which can be seen more clearly in Table 1 and 2. For further quantitative analysis, we report the means and variances of the ﬁnal test errors on both datasets in Table 1 and 2, respectively. Both ﬁgures convey the following two messages. Firstly, on both datasets with ResNet, SGD+Momentum is uniformly better than Adam in that it always converges to a solution with lower test error.
Such advantage can be immediately seen by comparing the two row blocks within each column of both tables. It is also consistent with the common wisdom (Wilson et al., 2017). Secondly, for both Adam and SGD+Momentum, the NG adap version has the best generalization performance (which is mark bold in each column), while the gradient clipping is inferior to all the remaining variants. While the normalized SGD+Momentum is better than the vanilla version, Adam slightly outperforms its normalized counterpart.
Those observations are consistent across networks with variant depths [Figure] [Figure] [Table]
[Table]
4. R ESIDUAL N ETWORK FOR I MAGE N ET C LASSIFICATION
In this section [Table] we further test our methods on ImageNet 2012 classiﬁcation challenges, which consists of more than 1.2M images from 1,000 classes.
We use the given 1.28M labeled images for training and the validation set with 50k images for testing.
We employ the validation set as the test set, and evaluate the classiﬁcation performance based on top-1 and top-5 error.
The pre-activation (He et al., 2016b) version of ResNet is adopted in our experiments to perform the classiﬁcation task.
Like the previous experiment, we again compare the performance on SGD+Momentum and Adam.
We run our experiments on one GPU and use single scale and single crop test for simplifying discussion.
We keep all the experiments settings the same as the publicly available Torch implementation. That is, we apply stochastic gradient descent with momentum of 0.9, weight decay of 0.0001, and set the initial learning rate to 0.1.
The exception is that we use mini-batch size of 64 and 50 training epochs considering the GPU memory limitations and training time costs.
Regarding learning rate annealing, we use 0.001 exponential decay.
As for Adam, we search the initial learning rate in range { 0 . 0005 , 0 . 001 , 0 .
Other setups are the same as the SGD optimization framework. Due to the time-consuming nature of training the networks (which usually takes one week) in this experiment, we only test on a 34-layer ResNet and compare SGD and Adam with our default NG method on the testing error of the classiﬁcation.
From Table 3, we can see normalized gradient has a non-trivial improvement on the testing error over the baselines SGD and Adam.
Besides, the SGD+Momentum again outperforms Adam, which torch is consistent with both the common wisdom (Wilson et al., 2017) and also the ﬁndings in previous section.
[Table]
4. L ANGUAGE M ODELING WITH R ECURRENT N EURAL N ETWORK
Now we move on to test the algorithms on Recurrent Neural Networks (RNN).
In this section, we test the performance of the proposed algorithm on the word-level language modeling task with a popular type of RNN, i.e. single directional Long-Short Term Memory (LSTM) networks (Hochreiter &
Schmidhuber, 1997).
The data set under use is Penn Tree Bank (PTB) (Marcus et al., 1993) data, which, after preprocessed, contains 929k training words, 73k validation and 82k test words.
The vocabulary size is about 10k.
The LSTM has 2 layers, each containing 200 hidden units.
The word embedding has 200 dimensions which is trained from scratch.
The batch size is 100.
We vary the length of the backprop through time (BPTT) within the range { 40 , 400 , 1000 } .
To prevent overﬁtting, we add a dropout regularization with rate 0.5 under all the settings.
The results are shown in Figure 3.
The conclusions drawn from those ﬁgures are again similar to those in the last two experiments.
However, the slightly different yet cheering observations is that the AdamNG is uniformly better than all the other competitors with any training sequence length.
The superiority in terms of convergence speedup exists in both training and testing.
[Figure]
T r a i n
[Figure]
ss adamadamngadagradadagradngsgdmsgdmng
[Figure] [Figure] [Figure] T r a i n
[Figure]
ss adamadamngadagradadagradngsgdmsgdmng
[Figure]
[Figure]
s t
[Figure]
adamadamngadagradadagradngsgdmsgdmng
[Figure]
[Figure]
[Figure]
adamadamngadagradadagradngsgdmsgdmng
4. S ENTIMENT A NALYSIS WITH C ONVOLUTION N EURAL N ETWORK
The task in this section is the sentiment analysis with convolution neural network.
The dataset under use is Rotten Tomatoes 5 Pang & Lee (2005), a movie review dataset containing 10,662 documents, with half positive and half negative.
We randomly select around 90% for training and 10% for validation.
The model is a single layer convolution neural network that follows the setup of (Kim, 2014).
The word embedding under use is randomly initialized and of 128-dimension.
For each algorithm, we run 150 epochs on the training data, and report the best validation accuracy in
Table 4.
The messages conveyed by the table is three-fold.
Firstly, the algorithms using normalized gradient achieve much better validation accuracy than their unnormalized versions.
Secondly, those with adaptive stepsize always obtain better accuracy than those without.
This is easily seen by the comparison between Adam and SGDM.
The last point is the direct conclusion from the previous two that the algorithm using normalized gradient with adaptive step sizes, namely AdamNG, outperforms all the remaining competitors.
[Table]
C ONCLUSION
In this paper, we propose a generic algorithm framework for ﬁrst order optimization.
It is particularly effective for addressing the vanishing and exploding gradient challenge in training with non-convex loss functions, such as in the context of convolutional and recurrent neural networks.
Our method is based on normalizing the gradient to establish the descending direction regardless of its magnitude, and then separately estimating the ideal step size adaptively or constantly.
This method is quite general and may be applied to different types of networks and various architectures.
Although the primary application of the algorithm is deep neural network training, we provide a convergence for the new method under the convex setting in the appendix.
Empirically, the proposed method exhibits very promising performance in training different types of networks (convolutional, recurrent) across multiple well-known data sets (image classiﬁcation, natural language processing, sentiment analysis, etc.).
In general, the positive performance differential compared to the baselines is most striking for very deep networks, as shown in our comprehensive experimental study.
A PPENDIX
As a concrete example for Algorithm 1, we present a simple modiﬁcation of AdaGrad using blockwise normalized stochastic gradient in Algorithm 2, where g 1: t is a matrix created by stacking g 1 , g 2 ,... and g t in columns and g 1: t,j ∈ R t represents the j th row of g 1: t for j = 1 , 2 , . . .
, d .
Assuming the function F is convex over x for any ξ and following the analysis in (Duchi et al., 2011), it is straightforward to show a O ( 1 √ T ) convergence rate of this modiﬁcation.
In the following, we denote (cid:107) x (cid:107) W := √ x (cid:62) Wx as the the Mahalanobis norm associated to a d × d positive deﬁnite matrix W .
The convergence property of Block-Normalized AdaGrad is presented in the following theorem.
Sample the stochastic g it =
F
[Equation]
i ( x t ,ξ t ) (cid:107) F
[Equation]
x
[Equation]
) (cid:107)
Let g t = ( g 1 t , g 2 t , . .
, g Bt ) , g 1: t =
[ g 1 , g 2 , . . . , g t ] and s t =
( (cid:107) g 1: t, 1 (cid:107) 2 , (cid:107) g 1: t, 2 (cid:107) 2 , . . .
, (cid:107) g 1: t,d (cid:107) 2 ) 5:
Partition s t =
( s 1 t s 2 t , . . .
, s Bt ) in the same way as g t .
Let τ t = ( τ 1 t , τ 2 t , . .
, τ Bt ) with 6 τ it =
η (cid:107) F (cid:48)
i ( x t , ξ t ) (cid:107)
δ 1 d i + s it ) − 1 .
7: x t +1 = x
t − τ t ◦ g t 8: end for
Theorem 1 Suppose F is convex over x , (cid:107) F (cid:48)
( x t , ξ t ) (cid:107) 2 ≤ M and (cid:107) x
t − x ∗ (cid:107) ∞ ≤ D ∞ for all t for some constants M and D ∞ > 0 in Algorithm 2.
Let H t =
δI d + diag ( s t ) and H it =
δI d i + diag ( s it ) for t = 1 , 2 , . . .
and ¯ x T := 1 T (cid:80) Tt =1 x t .
Algorithm 2 guarantees
[Equation]
T
[Equation]
Proof: It is easy to see that H t is a positive deﬁnite and diagonal matrix.
According to the updating scheme of x t +1 and the deﬁnitions of H t , τ t and g t , we have
[Equation]
The inequality above and the convexity of F ( x, ξ ) in x imply
[Equation]
Taking expectation over ξ t for t =
[Equation]
[Equation]
and averaging the above inequality give
[Equation]
where we use the fact that (cid:107) F (cid:48) ( x t , ξ t ) (cid:107) 2 2 ≤ M 2 in the second inequality.
According to the equation (24) in the proof of Lemma 4 in (Duchi et al., 2011), we have
T
[Equation]
where the second inequality is due to Cauchy-Schwarz inequality and the fact that (cid:107) g
it (cid:107) 2 = 1 .
Following the analysis in the proof of Theorem 5 in (Duchi et al., 2011)
[Equation]
we show that
[Equation]
After applying (4) and (5) to (3) and reorganizing terms
[Equation]
we have
[Equation]
where the second inequality is because (cid:104) s T , [Equation] =
[Equation]
2 ≤ √ TBd which holds due to
Cauchy-Schwarz inequality and the fact that (cid:107) g
it (cid:107) 2 = 1 .