H YPER -R EGULARIZATION : A F RAMEWORK OF A DAPTIVE C HOICE FOR THE L EARNING R ATE IN G RADIENT D ESCENT
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a novel approach for adaptively selecting the learning rate in gradient descent methods.
Speciﬁcally, we impose a regularization term on the learning rate via a generalized distance, and cast the joint updating process of the parameter and the learning rate into a maxmin problem.
Some existing schemes such as AdaGrad (diagonal version) and WNGrad can be rederived from our approach.
Based on our approach, the updating rules for the learning rate do not rely on the smoothness constant of optimization problems and are robust to the initial learning rate.
We theoretically analyze our approach in full batch and online learning settings, which achieves comparable performances with other ﬁrst-order gradientbased algorithms in terms of accuracy as well as convergence rate.
I NTRODUCTION
The automatic choice of the learning rate remains crucial in improving the efﬁciency of gradient descent algorithms, especially for solving nonconvex optimization problems.
It is desirable to adaptively update the learning rate during the training process with a certain strategy.
The convergence guarantees of such a strategy in theories usually require the Lipschitz constant or smoothness constant of the objective function to be explicitly known (Nesterov, 2013; Bubeck et al., 2015), which is inaccessible in most cases, e.g., in deep neural networks.
Using the received gradient information to adjust the current learning rate is a natural approach.
In particular, the Steepest Descent uses the received gradient direction and an exact or inexact line search to obtain proper learning rates.
Another important idea is to approximate second-order methods like Newton method (Nocedal & Wright, 2006) and Quasi-Newton Methods (Liu & Nocedal, 1989).
Along this idea, for example, the Barzilai-Borweinin (BB) method (Barzilai & Borwein, 1988) in classical optimization and AdaGrad (Duchi et al., 2011) in online learning have been proposed.
In this paper we propose a novel framework to learn the learning rate that we call HyperRegularization .
More speciﬁcally, we regard the learning rate as a hyperparameter and cast its adaptive choice with the parameter training into a joint process.
We formulate this process as a maxmin framework by imposing a regularizer on the hyperparameter.
Furthermore, we demonstrate that AdaGrad and WNGrad (Wu et al., 2018) can be derived using a streamlined scheme based on
Hyper-Regularization.
In addition to solving the saddle point problem exactly, we also provide an alternating strategy to solve the problem approximately.
We respectively give theoretical analysis for these two updating rules in full batch setting and online learning setting.
Speciﬁcally, our results of runtime bounds in full batch setting and regret bounds in online learning setting are comparable to the best known bound in corresponding settings and indicate our algorithms converge for any initial learning rate.
1. RELATED WORK
Steepest Descent uses the received gradient direction and an exact or inexact line search to obtain proper learning rates.
Although Steepest Descent uses the direction that descends most and the best learning rate that gives the most function reduction, Steepest Descent may converge very slow for convex quadratic functions when the Hessian matrix is ill-conditioned (see, Yuan, 2008).
In practice, some line search conditions such as Goldstein conditions or Wolfe conditions (see, Fletcher, 2013) can be applied to compute the learning rate.
In online or stochastic settings, one observes stochastic gradients rather than exact gradients and line search methods become less effective.
The Barzilai-Borwein method (Barzilai & Borwein, 1988) which was motivated by quasi-Newton methods presents a surprising result that it could lead to superlinear convergence in convex quadratic problem of two variables.
Although numerical results often show the Barzilai-Borwein method converges superlinearly in solving nonlinear optimization problem, no superlinear convergence results have been established even for an n -dimensional strictly convex quadratic problem with the order n > 2 (Barzilai & Borwein, 1988; Dai, 2013).
In minimizing the sum of cost functions and stochastic setting, SGD-BB proposed by Tan et al.
(2016) takes the average of the stochastic gradients in one epoch as an estimation of the full gradient.
But this approach can not directly be applied to online learning settings.
In online convex optimization (Zinkevich, 2003; Shalev-Shwartz et al., 2012; Hazan et al., 2016),
AdaGrad (Duchi et al., 2011) adapts the learning rate on per parameter basis dynamically.
Intuitively,
AdaGrad constructs approximation to the Hessian with diagonal of accumulated outer products of gradients.
This leads to many variants such as RMSProp (Tieleman & Hinton, 2012), AdaDelta (Zeiler, 2012), Adam (Kingma & Ba, 2015), etc.
Additionally, Cruz (2011) analyzed Adaptive Stochastic Gradient Descent (ASGD) which is a generalization of Kesten’s accelerated stochastic approximation
algorithm
(Kesten et al., 1958) for the high-dimensional case.
ASGD uses a monotone decreasing function with respect to a time variable to get learning rates.
Recently, Baydin et al.
(2018) proposed Hyper-Gradient Descent to learn the global learning rate in SGD, SGD with Nesterov momentum and Adam.
Hyper-Gradient Descent can be viewed as an approximate line search method in the online learning setting and it uses the update rule for the previous step to optimize the leaning rate in the current step.
However, HyperGradient Descent has no theoretical guarantee.
It is worth mentioning that Gupta et al.
(2017) proposed a framework named Uniﬁed Adaptive
Regularization from which AdaGrad and Online Newton Step (Hazan et al., 2007) can be derived.
However, Uniﬁed Adaptive Regularization gives an approach for approximating the Hessian matrix in second order methods.
1. N OTATION
Before introducing our approach, we present the notation.
We denote the set { x > 0 : x ∈ R } by R ++ .
For two vectors a , b ∈ R d , we use a / b to denote element-wise division, a ◦ b for elementwise product (the symbol ◦ will be omitted in the explicit context), a n = ( a n 1 , a n 2 , . . .
, a nd ) , and a ≥ b if a j ≥ b j for all j .
Let 1 be the vector of ones with an appropriate size, and diag( β ) be a diagonal matrix with the elements of the vector β on the main diagonal.
In addition, we deﬁne (cid:107) a [Equation] a , A a [Equation]
Given a set X ⊆ R d , a function f : X → R is said to satisfy f ∈ C 1 , 1 L ( X ) if f is continuously differentiable on X , and the derivative of f is Lipschitz continuous on X with constant L :
[Equation]
More general deﬁnition can be found in Nesterov (2013).
1. P ROBLEM S TATEMENT
For an online learning problem, a learner faces a sequence of convex functions { f t } with the same domain X ⊆ R d , receives (sub)gradient information g t ∈ ∂f t ( x t ) at each step t , and predicts a point x t +1 ∈ X .
Our theoretical analysis is based on two settings: full batch setting and online learning setting.
A full batch setting (or optimization setting) is to optimize a certain function F with exact gradient at each step, i.e., f t = F .
In this setting we assume F ∈ C 1 , 1 L
but it is not necessarily convex.
We analyze the convergence of our algorithms by capping the runtime T such that the minimum value of the norm of received gradients so far is less than a given error accuracy ε , that is,
[Equation]
In online learning settings, our analysis follows from Duchi et al.
(2011) and Kingma & Ba (2015).
We only assume that f t ’s are convex and try to give an upper bound for the regret
[Equation]
H YPER -R EGULARIZATION
Following the setting in AdaGrad (Duchi et al., 2011), we consider a generalization of the standard (sub)gradient descent as
[Equation]
This procedure can be viewed as the minimization problem:
[Equation]
To derive our hyper-regularization approach, we then formulate this minimization problem as a saddle point problem by adding a hyper-regularizer about the difference between the new learning rate β and an auxiliary vector η t .
Accordingly, we have
[Equation]
where D ( β , η ) is deﬁned as our hyper-regularizer and B t is a subset in R d .
We solve the saddle point problem for new predictor and new learning rate.
Our framework stems from the work of Daubechies et al.
(2010), where the authors adjust the weights of the weighted least squares problem by solving an extra objective function which added a regularizer about the weights to origin objective function.
The following subsections will explain some details about our framework.
2. T HE ϕ - DIVERGENCE
It is reasonable to choose a distance function to measure the difference between β and η .
In this paper, we use the ϕ -divergence 1 as our hyper-regularizer.
Deﬁnition 1 ( ϕ -divergence) .
Let ϕ : R ++ → R be a differentiable strongly convex function in R ++ such that ϕ (1) = ϕ (cid:48)
(1) = 0 , where ϕ (cid:48) is the derivative function of ϕ .
For such a function ϕ , the function D ϕ : R d ++ × R d ++ → R , which is deﬁne by
[Equation]
is referred to as the ϕ -divergence.
Remark.
Note that convex function ϕ with ϕ (1) =
ϕ (cid:48) (1) = 0 satisﬁes ϕ ( z ) ≥ 0 for all z > 0 , thus D ϕ ( u , v ) ≥ 0 for all u , v ∈ R d ++ , with equality iff u = v .
Remark.
For any convex function f , ϕ ( z ) =
f ( z ) − f (cid:48) (1)( z − 1) − f (1) is a proper function for our ϕ -divergence.
Remark.
In our framework, in order to solve the problem (4) feasibly, we always assume that lim
z ∞ ϕ (cid:48) ( z ) =
+ ∞ .
If one only requires ϕ to be convex and ϕ (1) = 0 , the resulting distance function D ϕ is called a f -divergence (Liese & Vajda, 1987; 2006).
The f -divergence has been widely applied in statistical machine learning (e.g., Nguyen et al., 2009).
Using the ϕ -divergence as our hyper-regularizer, we can rewrite the problem (4) as
[Equation]
The form of problem (6) implies that we can solve the problem for each dimension separately, and only a few extra calculations are required for each step.
2. MAX - MIN OR MIN - MAX
Consider a problem similar to (5),
[Equation]
x [Equation] [Equation] The solution of problem (5) is same as (7) in unconstrained case 2 , i.e. X = R d , B t =
R d ++ . However, if we set B t = [ b t, 1 , B t, 1 ] × · · · × [ b t,d , B t,d ] to constrain the range of β t +1 and suppose X = R d , the solution of (7) is more difﬁcult to get from the solution of the unconstrained problem, while the solution of (5) can be easily obtained by clipping the solution of the unconstrained problem to required range (see Lemma 2).
Thus, we choose (5) as our basic problem.
Lemma 2.
Suppose that B t =
[ b t, 1 , B t, 1 ] × · · · × [ b t,d , B t,d ] , and X = R d .
Let β ∗ be the solution of unconstrained problem max β (min x Ψ t ( x , β )) .
Then the solution of problem max β ∈B t (min x Ψ t ( x , β )) is β j = min { max { β ∗ j , b t,j } , B t,j } , for j = 1 , · · · , d.
2. S ELECTION OF η t
There are many ways to choose the sequence { η t } :
• The sequence { η t } is chosen in advance, before our methods start the job.
For example, set 1) η t = η 1 ; 2) η t = η √ t + 1 1 , where η > 0 is a prespeciﬁed constant.
• η t can be obtained from other adaptive learning rate methods such as AdaGrad.
• Set η t = β t .
In this case, the hyper-regularizer is the penalty for the change between β t + and β t .
The ﬁrst two ways can be treated as a smoothing technique to stabilize the learning rate.
We want to ensure the learning rate sequence { β t } close to another learning rate sequence { η t } .
Although setting η t = β t is our main focus, we keep the sequence of { η t } to maintain this ﬂexibility.
2. I SOTROPIC H YPER -R EGULARIZATION
We now show a special form of our framework that only adaptively maintains a single scalar learning rate.
We modify Hyper-regularization so that β is optimized over the set B t ⊆ { θ 1 : θ ∈ R ++ } of all positive multiples of the vector 1 ∈ R d .
Let η t =
η t 1 , and rewrite problem (5) as
[Equation]
We prefer to use Isotropic Hyper-Regularization as smoothing technique in online learning settings.
U PDATE R ULES AND A LGORITHMS
In this section we present two update rules of our hyper-regularization framework.
The ﬁrst update rule is solving the saddle point problem (5) exactly.
That is,
[Equation]
The following lemma and Algorithm 1 give the concrete scheme of our iterations.
Lemma 3.
Considering problem (6) without constraints and solving the problem exactly, we get new predictor x t +1 and new learning rate β t +1 such that
[Equation]
Remark.
Note that AdaGrad (Duchi et al., 2011) and WNGrad (Wu et al., 2018) are special cases of Algorithm 1 with a particular choice of ϕ (detailed derivation in Appendix B).
• If ϕ ( z ) =
z + 1 z − 2 , then we can derive AdaGrad from Algorithm 1.
• If ϕ ( z ) = 1 z − log( 1 z ) − 1 , then we can derive WNGrad from Algorithm 1.
3. A LTERNATING U PDATE R ULE
However, it is sometimes difﬁcult to solve the equation (10), especially in the case where ϕ is a quadratic function (equation (10) will be a cubic equation in one variable for any j ).
In practice, using an alternating update strategy is more recommended.
Under the assumption that the optimal value of β is close to η t , we solve an approximate equation for β t +
[Equation]
and update the new predictor x t +1 using
[Equation]
Applying the alternating update rule under the same assumption in Lemma 3, we obtain the following lemma and Algorithm 2.
Lemma 4.
Considering problem (6) without constraint and following from the alternating update rule, we get new predictor x t +1 and new learning rate β t +1 as
[Equation]
Algorithm 2 GD with Hyper-regularization using alternating update rule
,j
[Equation] end Computing the inverse function of ϕ (cid:48) is usually easier than solving the equation (10) in practice, especially for the widely used ϕ -divergences (more details can be found in Appendix D.1).
Remark.
If η t = β t , the following simpliﬁed alternating rule can be employed:
[Equation]
We leave the corresponding algorithm 3 in Appendix C.
3. M ONOTONICITY
Before giving more analysis, let us show monotonicity of both the two update rules.
The monotonicity implies that only the property of convex function ϕ on interval [1 , + ∞ ) will inﬂuence the efﬁciency of our algorithms.
Lemma 5.
β t +1 obtained from equation (9) or (11) satisﬁes β t +1 ≥ η t .
When setting β t = η t , we have that β t +1 ≥ β t .
T HEORETICAL A NALYSIS
In this section we always set η t = β t and assume that x and β are unconstrained, i.e., X = R d and B t = R d
++ .
We ﬁrst discuss the convergence rate of the two update rules in full batch setting with assumption that the objective function F is L -smooth but not necessarily convex in Section 4.1.
Next we turn to online convex learning setting and establish a theorem about the regret bounds in Section 4.2.
Our results for both the settings show that our algorithms are robust to the choice of initial learning rates and do not rely on the Lipschitz constant or smoothness constant.
4. I SOTROPIC H YPER -R EGULARIZATION IN FULL BATCH SETTING
Recall that we set f t = F in the full batch setting, and assume that F ∈ C 1 , 1 L without convexity.
In this case, two update rules can be written as
[Equation]
Next we show that both update rules (13) and (14) are robust to the choice of initial learning rate.
Theorem 6.
Suppose that ϕ ∈ C 1 , 1 l ([1 , + ∞ )) , ϕ is α -strongly convex, F ∈ C 1 , 1 L ( R d ) , and F ∗ = inf x F ( x ) >
−∞ .
For any ε ∈ (0 , 1) , the sequence { x t } obtained from update rules (13) or (14) satisﬁes
[Equation]
after T = O (cid:0) 1 ε (cid:1) steps.
More detailed results of Theorem 6 for runtime can be found in Theorems 21 and 22 in Appendix I. Theorem 6 shows that both runtime of the two update rules can be bound as O (1 /ε ) for any constant L and initial learning rate β 0 .
As a comparison, in classical convergence result ((1.2.13) in Nesterov (2013) or Theorem 20 in Appendix), the upper bound of runtime is O (1 /ε ) only for a certain range (related to L ) of initial learning rates.
4. H YPER -R EGULARIZATION IN O NLINE L EARNING S ETTING
We now establish the result of convergence rate for Algorithms 1 and 2 in online convex learning, i.e., the f t are convex.
Especially, we try to bound regrets (1) by O ( √ T ) for Algorithms 1 and 2.
Theorem 7.
Suppose that ϕ ∈ C 1 , 1 l ([1 , + ∞ )) , and ϕ is α -strongly convex.
Assume that (cid:107) g t (cid:107) ∞ ≤
G , and (cid:107) x t − x ∗ (cid:107) ∞ ≤ D ∞ .
Then the sequence { x t } obtained from Algorithm 1 satisﬁes
[Equation]
22 , and the sequence { x t } obtained from Algorithm 2 (or 3) satisﬁes
[Equation]
Note that under the assumption in Theorem 7, (cid:80) dj
=1 (cid:107) g 0: T − 1 ,j (cid:107) 2 ≤ dG √ T , hence R ( T ) =
O ( √ T ) .
Our result is comparable to the best known bound for convex online learning problem (see Hazan et al., 2016; Duchi et al., 2011; Kingma & Ba, 2015).
E XPERIMENTS
In this paper our principal focus has been to develop a new approach for adaptive choice of the learning rate in ﬁrst-order gradient-type methods.
However, this new approach also brings some insights into the resulting algorithms.
Thus, it is interesting to conduct empirical analysis of the learning algorithms with different choice methods for the learning rate.
5. T HE S ET - UP
To derive a learning algorithm from the Hyper-Regularization framework, we have to ﬁrst give the ϕ divergences (Pardo, 2005).
Speciﬁcally, the algorithms from our framework in the following experiments are derived from the following four ϕ divergences (full implementations are displayed in the Appendix D.1):
• ϕ ( t ) =
t log t − t + 1 from KL-divergence inducing KL algorithm.
• ϕ ( t )
= − log t + t − 1 from Reverse KL-divergence inducing RKL algorithm.
• ϕ ( t ) =
( √ t − 1) 2 from Hellinger distance inducing Hellinger algorithm.
• ϕ ( t ) =
( t − 1) 2 from χ 2 distance inducing χ 2 algorithm.
As mentioned in Section 3, even with a ﬁxed ϕ divergence, the generated algorithm still varies with different update rules.
For simplicity, different update rules were compared in advance to select the speciﬁc one for any ϕ divergence in the following experiments.
To maintain stable performance, the technique of growth clipping is applied to all algorithms in our framework.
Actually, growth clipping fulﬁlls the constraints placed on the increasing speed of β t by B t in Lemma 2.
Speciﬁcally, β t +1 in our experiments falls in [ β t , 2 β t ] .
Detailed observations on how the β t of our algorithms increases are left in Appendix D.3.
Experiments involve the four algorithms generated above as well as other ﬁrst-order gradient-based algorithms including SGD (with no learning rate decay), SGD-BB, and Hyper-Gradient Descent algorithms.
These algorithms are evaluated on tasks of image classiﬁcation with a logistic classiﬁer on the databases of MNIST (LeCun et al., 2010) and CIFAR-10 (Krizhevsky & Hinton, 2009).
Initial learning rate (in the usual sense, i.e., 1 /β 0 ) varies from 10 − 3 to 10 1 for the test of convergence performance of these algorithms.
Experiments are run using Tensorﬂow (Abadi et al., 2016), on a machine with Intel Xeon E5-2680 v4 CPU, 128 GB RAM, and NVIDIA Titan Xp GPU.
5. U PDATE R ULE S ELECTION
Taking the RKL algorithm as an example, we refer to algorithms deduced from Algorithm 1, 2, and 3 as RKL 1 , RKL 2 , and RKL 3 .
We train a two-layer neural network with a hidden layer of 500 units on the MNIST database.
Experiments are in online learning setting with a batch size of 128, and (cid:96)regularization is applied with a coefﬁcient of 10 − 4 .
[Figure]
Figure 1 demonstrates the training loss and test accuracy of these algorithms with various initial learning rates.
After ﬁxing the learning rate with the least training loss, we compare their performances throughout the training process.
Sharing comparable performances at small initial learning rate with RKL 3 , RKL 1 and RKL 2 perform better at relatively large learning rate.
Generally speaking, Algorithm 1 often suffers a higher computation complexity than Algorithmfor the difﬁculty of getting an analytical solution.
Detailed observations are left in Appendix D.1.
Therefore, we apply the second update rule to our algorithms without explicit notiﬁcations.
5. F ULL B ATCH S ETTING AND O NLINE L EARNING S ETTING
We investigate our algorithms in the full batch setting on the MNIST database where algorithms receive the exact gradients of the objective loss function each iteration.
[Figure]
In terms of online learning setting, we train a VGG Net (Simonyan & Zisserman, 2014) with batch normalization on the CIFAR-10 database with a batch size of 128, and an (cid:96) 2 regularization coefﬁcient of 10 − 4 .
We as well perform data augmentation as He et al.
(2016) to improve the training.
Figure 2 and Figure 3 show the convergence performances on both settings, respectively.
Achieving general comparable performances with other ﬁrst-order gradient-based algorithms, our algorithms outperform most of other algorithms at risky large initial learning rate.
Even at the ﬁxed initial learning rate with the least training loss, the performances of our algorithms still achieve the same training performances with others. [Figure]
D ISCUSSION
As a supplement, we point out that logarithmic regret bounds under assumption f t is strongly convex, like Hazan et al. ; Mukkamala & Hein (2017), can be established with a different class of distance function. We leave the details in Appendix H. In this paper
[Figure] we propose a novel framework distinct from previous main approaches like line search and approximate second-order methods. It worth noting that Hyper-Regularization can generates efﬁcient algorithms for optimization problems with regularization terms shown above and we expect new ideas of more efﬁcient terms in the future [Figure]
A S OLUTION E XISTENCE
Note that the function h ( β ) =
β 2 ϕ (cid:48)
( β/η t,j ) is an increasing continuous function and
lim z
→ + ∞ ϕ (cid:48) ( z ) =
+ ∞ ϕ (cid:48) (1) = 0 , so [0 , + ∞ ) is a subset of the range of h ( β ) and the solution of (10) exists.
For the same reason, the solution of (12) exists.
B S PECIAL C ASES OF A LGORITHM
In this section, We will point out that Adagrad (Duchi et al., 2011) and WNGrad (Wu et al., 2018) are special cases of Algorithm 1.
If we set ϕ ( z ) =
z + 1 z − 2 , then the new learning rate β t +1 can be obtained by
[Equation]
that implies,
[Equation]
and we drive AdaGrad from Hyper-Regularization.
Similarly, we can get WNGrad by setting ϕ
( z ) =
z − log 1 z − 1 .
In fact, β t +1 employs update
[Equation]
on the other words,
[Equation]
i.e., the update rule of WNGrad.
C S IMPLIFIED A LTERNATING U PDATE R ULE
Algorithm 3 GD with Hyper-regularization using simpliﬁed alternating update rule and η t = β t
Input: β 0 > 0 , x
The weakness of the simpliﬁed alternating update rule is that we get the new predictor x t +1 by β t which is unrelated to current gradient g t .
The regret bound of Algorithm 3 has been shown in
Theorem 7.
D S UPPLEMENTARY FOR N UMERICAL E XPERIMENTS
Full versions of Hellinger, KL, RKL, and χ 2 algorithms through Algorithm 1, 2, and 3 are listed below.
Apparently, it is of great complexity to compute an analytical solution for β t +1 for the equations marked red.
D. F ULL V ERSIONS OF G ENERATED A LGORITHMS
KL(cid:26) β 2 t +1 log( β t +1 / β t ) =
g 2 t x t +
= x
t − g t / β t +
KL(cid:26) β t +1 = β t exp( g 2 t / β 2 t ) x t +1 = x
t − g t / β t +
KL(cid:26) x t +1 = x
t − g t / β t β t +1 = β t exp( g 2 t / β 2 t )
RKL(cid:40)
β t +1 = 12 (cid:16) β t
+ (cid:112) β 2 t
+ 4 g 2 t (cid:17) x t +1 = x
t − g t / β t +
RKL(cid:26) β t +1 = β 3 t / ( β 2 t − g 2 t ) x t +1 = x
t − g t / β t +
RKL(cid:26) x
t +1 = x
t − g t / β t β t +1 = β 3 t / ( β 2 t − g 2 t )
H(cid:26) β 2 t +1 (1 − (cid:112) β t / β t +1 ) =
g 2 t x t
+
=
x
t − g t / β t +
H(cid:26) β t +1 = β 5 t / ( β 2 t − g 2 t )x t +1 = x
t − g t / β t +
H(cid:26) x t +1 = x
t − g t / β t β t +1 = β 5 t / ( β 2 t − g 2 t )χ(cid:26) 2 β 2 t +1 ( β t +1 / β t − 1) =
g 2 t x t
+
=
x
t − g t / β t + χ 2(cid:26) β t
+1 = β t (cid:0) 1 + g 2 t / (2 β 2 t ) (cid:1) x
t +1 = x
t − g t / β t + χ(cid:26) x t +
= x
t − g t / β t β t +1 = β t (cid:0) 1 + g 2 t / (2 β 2 t ) (cid:1)
D. V ERSION C OMPARISON ON O THER A LGORITHMS
Figure 4, 5 and 6 shows the results of comparison between versions of KL, Hellinger
[Figure]
and χ 2 algorithms on the base of MNIST in the online learning setting
[Figure] [Figure]
[Figure]
D. F IGURES ON β t I NCREASING P ROCESS
All the following experiments are based on the MNIST database in online learning setting with a batch size of 128.
For simplicity, all initial β 0 s are ﬁxed to be 10 − 0 .
Considering the sparse feature of parameters in this task, ﬁgure 7 only observe the changing process of the maximum of β t .
[Figure]
E P ROOF OF L EMMA
Proof.
First, it is trivial to get
[Equation]
The partial derivative of Ψ t, x ( β ) with respect to β j is
[Equation]
Note that ϕ is a convex function, so ϕ (cid:48) is a non-decreasing function, and ∂ Ψ t, x ( β ) ∂β
j is a non-increasing function.
Recall that β ∗ be the solution of unconstrained problem max β (min x Ψ t ( x , β )) , hence, β ∗ j is a zero of function ∂ Ψ t, x ( β ) ∂β j .
Moreover, if β ∗ j > B t,j , we have ∂ Ψ t, x ( β ) ∂β j ≥ 0 .
Thus, Ψ t, x ( β ) with respect to β j is a nondecreasing function, and arg max β j Ψ t, x ( β ) =
B t,j .
For a similar reason, if β ∗ j < b t,j , then arg max β j Ψ t, x ( β )
=
b t,j .
In conclusion,
[Equation]
F P ROOF OF L EMMA
In this section, we denote that Ψ t, x ( β ) =
min x ∈X Ψ t ( x , β ) .
Lemma 8.
β t +1 obtained from equation (9) satisﬁes β t +1 ≥ η t .
Proof.
Recall that ϕ (1) =
ϕ (cid:48) (1) = 0 , so ϕ ( x ) ≥ 0 for all x and D ϕ ( β, η t ) =
η t ϕ ( β/η t ) ≥ 0 .
If β < η t , then for all x ∈ X
[Equation]
[Equation]
Hence, min x ∈X Ψ t ( x , β ) < min x ∈X Ψ t ( x , η t ) , i.e., Ψ t, x ( β ) <
Ψ t, x ( η t ) .
It means β t +1 = arg max β ∈B Ψ t, x ( β ) ≥ η t .
Lemma 9.
β t +1 obtained from equation (11) satisﬁes β t +1 ≥ η t .
Proof.
Let y
=
arg
min x Ψ( x , η t ) .
If β < η t , then
[Equation]
=
[Equation]
Hence β t +1 = arg max β ∈B Ψ t ( y , β ) ≥ η t .
G C ONVERGENCE RATES IN ONLINE LEARNING SETTING
Recall the deﬁne of regret
[Equation]
(15) where x ∗ =
arg min x ∈X (cid:80) T − 1 t =
f t ( x ) .
We show our hyper-regularizer method with three update rules (Algorithm 1, 2 and 3) have O ( √ T ) regret bounds.
Lemma 10 (Lemma 4 in Adagrad) .
Consider an arbitrary real-valued sequence { a i } and its vector representation a 1: i = ( a 1 , · · · , a i ) (cid:62) .
Then
[Equation] [Equation]
holds.
Proof.
Let us use induction on T to prove inequality (10).
For T = 1 , the inequality trivially holds.
Assume the bound (16) holds true for T − 1 , in which case
[Equation]
We denote b T =
(cid:80) Tt =1 a 2 t and have
[Equation]
Lemma 11.
Suppose the sequence { x t } and sequence { β t } satisfy x t +1 = x
t − g t / β t +1 .
Then the regret satisﬁes
[Equation]
Proof.
Note that
[Equation]
and
[Equation]
i.e
[Equation] [Equation]
Hence
[Equation]
≤
[Equation]
Lemma 12.
Suppose an increasing function ψ satisﬁes ψ (1) = 0 and ψ ( x ) ≤ l ( x − 1) .
Consider a real valued sequence { g t } t =0: T − 1 and a positive sequence { β t } t =0: T which satisﬁes | g t | ≤ G , β 2 t +1 ψ (cid:16) β t +1 β t (cid:17)
=
g 2 t , t = 0 , · · · , T − 1 , β 0 ≥ 0 .
We can bound β T as
[Equation]
where c = (cid:113)
β 20 β 20 +2 G 2 /l .
Moreover, we have
[Equation]
Remark.
We point out that
• β t +1 ≥ β t (If β t +1 < β t , then β 2 t +1 ψ ( β t +1 /β t ) < 0 ≤ g 2 t ), • β t +1 is unique with respect to β t due to the fact that the function ˆ ψ ( β )
=
β 2 ψ ( β/β t ) is strictly increasing.
Proof.
Assume that β t ≥ c
[Equation]
, where c > 0 is a variable coefﬁcient.
Let us ﬁnd out a speciﬁc c such that
[Equation]
Note that
[Equation]
Deﬁne a cubic polynomial
[Equation]
and h is an increasing function when β ≥ β t .
If h (cid:18)
[Equation]
, according to h ( β t +1 ) ≥ 0 , then β t +1 ≥ c
[Equation]
g 2 i
[Equation]
Denote b = β 20 + 2 l (cid:80) t − 1 i =
g 2 i .
So we just need to choose c such that
[Equation]
where the ﬁrst inequality holds for the assumption β t ≥ c (cid:113) β 20 + 2 l (cid:80) t − 1 i =
g 2 i , or
[Equation]
or
[Equation]
Thus, c just need to satisfy
[Equation]
According to b ≥ β 20 , g 2 t ≤ G 2 , hence
[Equation]
So if we choose c = (cid:113) β 20 β 20 +2 G 2 /l , then β 1 > β 0 > cβ 0 , hence β t ≥
[Equation]
Moreover, following from Lemma 10, we have
[Equation]
g
[Equation]
Lemma 13.
Suppose an increasing function ψ satisﬁes ψ (1) = 0 and ψ ( x ) ≤ l ( x − 1) .
Consider a real valued sequence { g t } t =0: T − 1 and a positive sequence { β t } t =0: T which satisﬁes | g t | ≤ G , β 2 t ψ (cid:16) β t +1 β t (cid:17)
=
g 2 t , t = 0 , · · · , T − 1 , β 0 ≥ 0 .
We can bound β T as
[Equation]
Moreover, we have
[Equation] [Equation]
Proof.
Same as inequality (20), we have
[Equation]
hence
[Equation]
Furthermore, following from Lemma
[Equation]
we have
[Equation]
Theorem 14.
Suppose that ϕ ∈ C 1 , 1 l ([1 , + ∞ ))
[Equation]
and ϕ is α -strongly convex.
Assume that (cid:107) g t (cid:107) ∞ ≤
G
[Equation]
and (cid:107) x
t − x ∗ (cid:107) ∞ ≤ D ∞ .
Then the sequence { x t } obtained from Algorithm 1 satisﬁes
[Equation]
Proof.
Following from Lemma 11,
[Equation]
≤
[Equation]
≤
[Equation]
Recall ϕ is a α -strongly convex function, so,
[Equation]
and
[Equation]
The function ψ = ϕ (cid:48) satisﬁes ψ (1) = 0 and ψ ( x ) ≤ l ( x − 1) according to the smoothness of ϕ .
Following from Lemma
[Equation]
we have
[Equation]
Combining inequality (23) and (24), we have
[Equation]
Theorem 15.
Suppose that ϕ ∈ C 1 , 1 l ([1 , + ∞ )) , and ϕ is α -strongly convex.
Assume that (cid:107) g t (cid:107) ∞ ≤
G , and (cid:107) x t − x ∗ (cid:107) ∞ ≤ D ∞ .
Then the sequence { x t } obtained from Algorithm 2 (or 3) satisﬁes
[Equation]
Proof.
Similar to the proof of Theorem 14, for Algorithm 2, we have
[Equation]
With same reason
[Equation]
for Algorithm
[Equation]
we have
[Equation]
Note that for both algorithms
[Equation]
holds, so
[Equation]
Thus
[Equation]
following from Lemma
[Equation]
for both Algorithm 2 and
[Equation]
we have
[Equation]
H L OGARITHMIC B OUNDS
In this section, we will use a different class of ‘distance’ function for problem 4, and establish logarithmic regret bounds under assumption f t is strongly convex.
Our analysis and proof follow from Hazan et al.
; Mukkamala & Hein (2017).
First, we deﬁne µ -strongly convexity.
Deﬁnition 16 (Deﬁnition 2.1 in (Mukkamala & Hein, 2017)) .
Let X ⊆
R
d be a convex set.
We say that a function f : X → R is µ -strongly convex, if there exists µ ∈ R d with µ j > 0 for j = 1 , · · · , d such that for all x , y ∈ X ,
[Equation]
Let ξ =
min j =1: d µ j , then this function is ξ -strongly convex (in the usual sense), that is
[Equation]
The modiﬁcation SC-Hyper-Regularization of Hyper-Regularization which we propose in the following uses a family of distance function D : R d ++ × R d ++ → R formulated as
[Equation]
25) where ϕ is convex function with ϕ (1) =
ϕ (cid:48) (1) =
like we used in ϕ -divergence.
Remark.
Same as ϕ -divergence, D ( u , v ) ≥ 0 for any u , v ∈ R d ++ .
Different from Algorithm 1 and 2, we add a hyper-parameter λ > 0 like AdaGrad to SC-HyperGradient.
Rewrite problem (5) as
[Equation]
) and corresponding two algorithms as
Algorithm 4 GD with SC-Hyper-regularization
Input
Receive
(cid:48)
( β β
) =
2 t,j , j
· · , d ; 5:
; 6: end for
Input t +1 ,
= β t,j
) −
2 t, λβ
, d ; 5: t
1 ; 6: end for
Remark.
Same as Lemma 5, the monotonicity of Algorithm 4 and 5 also holds.
Theorem 17.
Suppose that f t is µ -strongly convex for all t , ϕ ∈ C 1 , 1 l ([1 , + ∞ )) , and ϕ is αstrongly convex.
Assume that (cid:107) g t (cid:107) ∞ ≤ G , and λ ≥ G 2 / ( α min j =1: d µ j ) .
Then the sequence { x t } obtained from Algorithm 4 satisﬁes
[Equation]
and the sequence { x t } obtained from Algorithm 5 satisﬁes
[Equation]
Remark.
Under assumption in Theorem 17, we have (cid:107) g 0: T − 1 ,j (cid:107) 2 2 ≤ G 2 T , so R ( T ) =
O (ln( T )) .
To prove Theorem 17, we ﬁrst prove following lemma.
Lemma 18.
For an arbitrary real-valued sequence { a i } and a positive real number b ,
T (cid:88)
[Equation] [Equation]
Proof.
Let b
= b, b t =
b + (cid:80) ti =1 a 2 i , t ≥ 1 , then
T (cid:88)
[Equation] [Equation]
(cid:90)
[Equation] [Equation]
Like Lemma 12 and 13, similar lemma holds for Algorithm 4 and 5.
Lemma 19.
Suppose an increasing function ψ satisﬁes ψ (1) = 0 and ψ ( x ) ≤ l ( x − 1) .
Consider a real valued sequence { g t } t =0: T − 1 and a positive sequence { β t } t =0: T which satisﬁes | g t | ≤ G , β 0 > 0 .
If ( β 2 t +1 /β t ) ψ ( β t +1 /β t ) =
g 2 t , t = 0 , · · · , T − 1 , then we have
[Equation]
(28) and
[Equation]
Meanwhile, if β t ψ ( β t +1 /β t ) =
g 2 t , t = 0 , · · · , T − 1 , then we have β t ≥ β 0 + 1 l t
[Equation]
1 (cid:88) i =
g 2 i , t = 1 , · · · , T (30) and
T − 1 (cid:88) t = g
[Equation]
l
[Equation]
(cid:32)
[Equation]
(cid:80)
[Equation]
−
[Equation]
=
[Equation]
lβ
[Equation]
(cid:33) .
Proof.
Using same methods in proof of Lemma 12 and 13, the conclusion can be deduced from
Lemma 18 easily.
proof of Theorem 17 .
Like Lemma 11, in strongly convex case, we have
[Equation] [Equation]
≤
[Equation]
Note that in Algorithm 4,
[Equation]
holds, and in Algorithm 5, same conclusion holds:
[Equation]
Hence, if λ ≥ max j =1: d G 2 αµ j , then β t +1 − β t ≤ µ , and
T − 1 (cid:88) t =
[Equation]
∗
[Equation]
≤ 0 .
On the other hand,
[Equation]
following from Lemma 19, we have
T − 1 (cid:88) t =
[Equation]
(cid:33) in Algorithm 4,
T − 1 (cid:88) t
= (cid:107)
[Equation]
≤
[Equation]
=
[Equation]
ln (cid:32)
+ (cid:107) g
[Equation]
(cid:107)λlβ
[Equation]
(cid:33) in Algorithm 5.
I C ONVERGENCE RATES IN F ULL BATCH S ETTING In
this section, we will discuss the convergence of our methods in full batch settings.
We ﬁrst review a classical result on the convergence rate for gradient descent with ﬁxed learning rate.
Theorem 20.
Suppose that F ∈ C 1 , 1 L ( R d ) and F ∗ =
inf x F ( x ) >
−∞ .
Consider gradient descent with constant step size, x t +
= x
t − ∇ F ( x t ) b .
If b > L 2 , then
[Equation]
after at most a number of steps
[Equation]
Proof.
Following from the fact that F is L -smooth, we have
[Equation]
When b > L 2 , 1 − L 2 b > 0 .
So
T
[Equation] [Equation]
and min
[Equation]
Remark.
If we choose b ≤ L 2 , then convergence of gradient descent with constant learning rate is not guaranteed at all.
Like Algorithm 3, another update rule is worth considering in full batch setting:
[Equation]
Next we will show that both update rules (13) and (14) are robust to the choice of initial learning rate.
Our proof is followed from the proof of Theorem 2.3 in WNGrad (Wu et al., 2018).
Note that in update rule (13), β t +1 satisﬁes
[Equation]
while in update rule (14) and (33), β t +1 satisﬁes
[Equation]
Theorem 21 (Convergence rate of update rule (13)) .
Suppose that ϕ ∈ C 1 , 1 l ([1 , + ∞ )) , ϕ is αstrongly convex, and F ∈ C 1 , 1 L ( R d ) , F ∗ =
inf x F ( x ) >
−∞ .
For any ε ∈ (0 , 1) , the sequence { x t } obtained from update rule (13) satisﬁes
[Equation]
, after T steps, where
[Equation]
(cid:17)(cid:17)
[Equation]
Theorem 22 (Convergence rate of update rule (14)) .
Suppose that ϕ ∈ C 1 , 1 l ([1 , + ∞ )) , ϕ is αstrongly convex, and F ∈ C 1 , 1 L ( R d ) , F ∗ =
inf x F ( x ) >
−∞ .
For any ε ∈ (0 , 1) , the sequence { x t } obtained from update rule (14) satisﬁes
[Equation]
after T steps, where
[Equation]
Theorem 23 (Convergence rate of update rule (33)) .
Suppose that ϕ ∈ C 1 , 1 l ([1 , + ∞ )) , ϕ is αstrongly convex, and F ∈ C 1 , 1 L ( R d ) , F ∗ =
inf x F ( x ) >
−∞ .
For any ε ∈ (0 , 1) , the sequence { x t } obtained from update rule (33) satisﬁes
[Equation]
after T steps, where
[Equation]
We begin our proof by following lemma
[Equation]
Lemma 24.
Suppose ϕ ∈ C 1 , 1 l ( R ++ ) .
Fix ε ∈ (0 , 1] .
In both update rules (13) and (14), after
T = (cid:24) log( Lβ 0 ) log( εlL 2 +1) (cid:25) + 1 steps, either min t =0: T − 1 (cid:107) g t (cid:107) 22 ≤ ε , or β T ≥ L holds.
Proof.
Assume that β T < L and min t =0: T − 1 (cid:107) g t (cid:107) 22 > ε .
Recall that the sequence { β t } is an increasing sequence.
Hence, β t < L for 0 ≤ t ≤ T .
So, for all 0 ≤ t ≤ T − 1 ,
[Equation]
Note that ϕ is a l -smooth convex function, and β t +1 /β
t ≥ 1 .
So
[Equation]
then
[Equation]
In this case,
[Equation]
holds
but it is impossible according to the setting of T in the lemma.
We ﬁrst prove Theorem 21 using following lemma.
Lemma 25.
In update rule (13), suppose F ∈ C 1 , 1 L ( R d ) , ϕ ∈ C 1 , 1 l ( R ++ ) , and ϕ is α -strongly convex function.
Denote F ∗ = inf x F ( x ) >
−∞ .
Let t 0 ≥ 1 be the ﬁrst index such that β t 0 ≥ L .
Then for all t ≥ t 0 ,
[Equation]
35) and moreover,
[Equation] [Equation]
Proof.
Same as equation (32),
[Equation]
For t ≥ t 0 − 1 , β t +1 ≥ L , so
[Equation]
Hence, for all k ≥ 0 ,
[Equation]
i.e.,
[Equation]
Note that ϕ is α -strongly convex and β
[Equation]
1 ϕ
[Equation]
and
[Equation]
Combining equation (38) and equation (39), we have
[Equation]
We remain to give an a upper bound for F ( x t 0 − 1 ) in the case t 0 > 1 .
Using equation (32) again, we get
[Equation]
In the above, the second inequality follows from the assumed l -smoothness of ϕ , and the last inequality follows from β t ≥ β 0 for all t ≥ 0 .
proof of Theorem 21 .
If t 0 = 1 , by equation (37), for all t ≥ 1 , we have
[Equation]
Then after T = 1 + (cid:108) 2( β
[Equation]
+2(
[Equation] [Equation]
F
[Equation] [Equation]
[Equation]
(cid:109) steps, min
[Equation]
Otherwise, if t 0 > 1 , we have β t 0 − 1 < L .
Then for all t ≥ t 0 ,
[Equation]
Denote the right hand of equation (40) as β max .
Using equation (37) again, for we have
[Equation]
Hence,
[Equation] [Equation]
At last, with recalling the conclusion of Lemma 24, after
[Equation]
steps, we have min t =
T − 1 (cid:107) g t (cid:107) 22 ≤ ε .
Next we prove Theorem 22.
Lemma 26.
In update rule (14), suppose F ∈ C 1 , 1 L ( R d ) , ϕ ∈ C 1 , 1 l ( R ++ ) , and ϕ is α -strongly convex function.
Denote F ∗ = inf x F ( x ) .
Let t 0 ≥ 1 be the ﬁrst index such that β t 0 ≥ L .
Then for all t ≥ t 0 ,
[Equation] [Equation]
and moreover,
[Equation]
Proof.
Same as the proof of Lemma 25, we ﬁrst get for all k ≥ 0 ,
[Equation]
Note that in update rule (14), β 2 t ϕ (cid:48) ( β t +1 /β t )
= (cid:107) g t (cid:107) 22 .
So
[Equation]
+ i
[Equation]
1 (cid:107)β t
[Equation]
i
[Equation]
If t 0 = 1 , then
[Equation]
and if t 0 ≥ 2 , then
[Equation]
At last, for t 0 > 0 , we have
[Equation]
proof of Theorem 22 .
The proof is completely similar to the proof of Theorem 21.
Next, we prove Theorem 23.
Lemma 27.
In update rule (33) [Equation] suppose F ∈ C 1 , 1 L ( R d ) , ϕ ∈ C 1 , 1 l ( R ++ ) , and ϕ is α -strongly convex function.
Denote F ∗ = inf x F ( x ) .
Let t 0 ≥ 0 be the ﬁrst index such that β t 0 ≥ L .
Then for all t ≥ t 0 ,
[Equation]
and moreover,
[Equation]
Proof.
Same as the proof of Lemma 25, we ﬁrst get
[Equation]
and
[Equation]
Note that in Algorithm 2, β 2 t ϕ (cid:48) ( β t +1 /β
t
[Equation]
= (cid:107) g t
[Equation]
So
[Equation]
At last, for t 0 > 0 , we have
[Equation]
Recall that
[Equation]
So if t 0 = 1 ,
[Equation]
And if t 0 ≥ 2 , we have
[Equation]
proof of Theorem 23 .
The proof is completely similar to the proof of Theorem
[Equation]