AI-SARAH: A DAPTIVE AND I MPLICIT S TOCHASTIC R ECURSIVE G RADIENT M ETHODS
Anonymous authors
Paper under double-blind review
A BSTRACT
We present AI-SARAH , a practical variant of SARAH .
As a variant of SARAH , this algorithm employs the stochastic recursive gradient yet adjusts step-size based on local geometry.
AI-SARAH implicitly computes step-size and efﬁciently estimates local Lipschitz smoothness of stochastic functions.
It is fully adaptive, tune-free, straightforward to implement, and computationally efﬁcient.
We provide technical insight and intuitive illustrations on its design and convergence.
We conduct extensive empirical analysis and demonstrate its strong performance compared with its classical counterparts and other state-of-the-art ﬁrst-order methods in solving convex machine learning problems
I NTRODUCTION
We consider the unconstrained ﬁnite-sum optimization problem
[Equation]
This problem is prevalent in machine learning tasks where w corresponds to the model parameters, f i ( w ) represents the loss on the training point i , and the goal is to minimize the average loss
P ( w ) across the training points.
In machine learning applications, (1) is often considered the loss function of Empirical Risk Minimization (ERM) problems.
For instance, given a classiﬁcation or regression problem, f i can be deﬁned as logistic regression or least square by ( x i , y i ) where x i is a feature representation and y i is a label.
Throughout the paper, we assume that each function f i , i ∈ [ n ] def = { 1 , ..., n } , is smooth and convex, and there exists an optimal solution w ∗ of (1).
1. M AIN C ONTRIBUTIONS
We propose AI-SARAH , a practical variant of stochastic recursive gradient methods (Nguyen et al., 2017) to solve (1).
This practical algorithm explores and adapts to local geometry.
It is adaptive at full scale yet requires zero effort of tuning hyper-parameters.
The extensive numerical experiments demonstrate that our tune-free and fully adaptive algorithm is capable of delivering a consistently competitive performance on various datasets, when comparing with SARAH , SARAH+ and other state-of-the-art ﬁrst-order method, all equipped with ﬁne-tuned hyper-parameters (which are selected from ≈ 5 , 000 runs for each problem).
This work provides a foundation on studying adaptivity (of stochastic recursive gradient methods) and demonstrates that a truly adaptive stochastic recursive algorithm can be developed in practice.
1. R ELATED W ORK
Stochastic gradient descent ( SGD ) (Robbins & Monro, 1951; Nemirovski & Yudin, 1983; ShalevShwartz et al., 2007;
Nemirovski et al., 2009; Gower et al., 2019) is the workhorse for training supervised machine learning problems that have the generic form (1).
In its generic form, SGD deﬁnes the new iterate by subtracting a multiple of a stochastic gradient g ( w t ) from the current iterate w t .
That is,
[Equation]
In most algorithms, g ( w ) is an unbiased estimator of the gradient (i.e., a stochastic gradient), E [ g ( w )]
= ∇ P ( w ) , ∀ w ∈ R d .
However, in several algorithms (including the ones from this paper), g ( w ) could be a biased estimator, and convergence guarantees can still be well obtained.
Adaptive step-size selection.
The main parameter to guarantee the convergence of SGD is the step-size .
In recent years, several ways of selecting the step-size have been proposed.
For example, an analysis of SGD with constant step-size ( α t = α ) or decreasing step-size has been proposed in Moulines & Bach (2011); Ghadimi & Lan (2013);
Needell et al.
; Nguyen et al.
Bottou et al.
Gower et al.
(2019; 2020b) under different assumptions on the properties of problem (1).
More recently, adaptive / parameter-free methods (Duchi et al., 2011; Kingma & Ba, 2015; Bengio, 2015; Li & Orabona, 2018; Vaswani et al., 2019;
Liu et al., 2019a;
Ward et al., 2019; Loizou et al., 2020) that adapt the step-size as the algorithms progress have become popular and are particularly beneﬁcial when training deep neural networks.
Normally, in these algorithms, the step-size does not depend on parameters that might be unknown in practical scenarios, like the smoothness parameter or the strongly convex parameter.
Random vector g ( w t ) and variance reduced methods.
One of the most remarkable algorithmic breakthroughs in recent years was the development of variance-reduced stochastic gradient algorithms for solving ﬁnite-sum optimization problems.
These algorithms, by reducing the variance of the stochastic gradients, are able to guarantee convergence to the exact solution of the optimization problem with faster convergence than classical SGD .
In the past decade, many efﬁcient variancereduced methods have been proposed.
Some popular examples of variance reduced algorithms are
SAG (Schmidt et al., 2017), SAGA (Defazio et al., 2014), SVRG (Johnson & Zhang, 2013) and SARAH (Nguyen et al., 2017).
For more examples of variance reduced methods, see Defazio (2016); Koneˇcný et al.
; Gower et al.
(2020a); Khaled et al.
; Horváth et al.
Among the variance reduced methods, SARAH is of our interest in this work.
Like the popular SVRG ,
SARAH algorithm is composed of two nested loops.
In each outer loop k ≥ 1 , the gradient estimate v 0 = ∇ P ( w k − 1 ) is set to be the full gradient.
Subsequently, in the inner loop, at t ≥ 1 , a biased estimator v t is used and deﬁned recursively as
[Equation]
where i ∈ [ n ] is a random sample selected at t .
A common characteristic of the popular variance reduced methods is that the step-size α in their update rule w t +1 =
w t − αv t is constant (or diminishing with predetermined rules) and that depends on the characteristics of problem (1).
An exception to this rule are the variance reduced methods with Barzilai-Borwein step size, named BB-SVRG and BB-SARAH proposed in Tan et al.
and Li & Giannakis (2019) respectively.
These methods allow to use Barzilai-Borwein ( BB ) step size rule to update the step-size once in every epoch; for more examples, see Li et al.
; Yang et al.
There are also methods proposing approach of using local Lipschitz smoothness to derive an adaptive step-size (Liu et al., 2019b) with additional tunable parameters or leveraging BB step-size with averaging schemes to automatically determine the inner loop size (Li et al., 2020).
However, these methods do not fully take advantage of the local geometry, and a truly adaptive algorithm: adjusting step-size at every (inner) iteration and eliminating need of tuning any hyper-parameters, is yet to be developed in the stochastic variance reduced framework.
This is exactly the main contribution of this work, as we mentioned in previous section.
M OTIVATION
With our primary focus on the design of a stochastic recursive algorithm with adaptive step-size, we discuss our motivation in this chapter.
A standard approach of tuning the step-size involves the painstaking grid search on a wide range of candidates.
While more sophisticated methods can design a tuning plan, they often struggle for efﬁciency and/or require a considerable amount of computing resources.
More importantly, tuning step-size requires knowledge that is not readily available at a starting point w 0 ∈ R d , and choices of step-size could be heavily inﬂuenced by the curvature provided ∇ 2 P ( w 0 ) .
What if a step-size has to be small due to a "sharp" curvature initially, which becomes "ﬂat" afterwards?
To see this is indeed the case for many machine learning problems, let us consider logistic regression for a binary classiﬁcation problem, i.e., f i ( w )
=
log(1 + exp( − y i
x Ti w ))
+ λ 2 (cid:107) w (cid:107) 2 , where x i ∈ R
d is a feature vector, y i ∈ {− 1 , +1 } is a ground truth, and the ERM problem is in the form of (1).
It is
[Figure]
easy to derive the local curvature of P ( w ) , deﬁned by its Hessian in the form
[Equation]
Given that a (1+ a ) 2 ≤ 0 .
25 for any a ≥ 0 , one can immediately obtain the global bound on Hessian, i.e. ∀ w ∈ R
d we have ∇ 2 P ( w ) (cid:22) 14 1 n (cid:80) ni
=1 x i x Ti + λI .
Consequently, the parameter of global
Lipschitz smoothness is L = 14 λ max ( 1 n (cid:80) ni
=1 x i x Ti ) + λ .
It is well known that, with a constant step-size less than (or equal to) 1 L , a convergence is guaranteed by many algorithms.
However, suppose the algorithm starts at a random w 0 (or at 0 ∈ R d ), this bound can be very tight.
With more progress being made on approaching an optimal solution (or reducing the training error), it is likely that, for many training samples, − y i x Ti w t (cid:28) 0 .
An immediate implication is that s i ( w t ) deﬁned in (3) becomes smaller and hence the local curvature will be smaller as well.
It suggests that, although a large initial step-size could lead to divergence, with more progress made by the algorithm, the parameter of local Lipschitz smoothness tends to be smaller and a larger step-size can be used.
That being said, such a dynamic step-size cannot be well deﬁned in the beginning, and a fully adaptive approach needs to be developed.
For illustration, we present the inspiring results of an experiment on real-sim dataset 1 with (cid:96) 2 -regularized logistic regression.
Figures 1(a) and 1(b) compare the performance of classical
SARAH with AI-SARAH in terms of the evolution of the optimality gap and the squared norm of recursive gradient.
As is clear from the ﬁgure, AI-SARAH displays a signiﬁcantly faster convergence per effective pass 2 .
Now, let us discuss why this could happen.
The distribution of s i as shown in Figured 1(d) indicates that: initially, all s i ’s are concentrated at 0 .
25 ; the median continues to reduce within a few effective passes on the training samples; eventually, it stabilizes somewhere below 0 .
Correspondingly, as presented in Figure 1(c), AI-SARAH starts with a conservative step-size dominated by the global
Lipschitz smoothness, i.e., 1 /λ max ( ∇ 2 P ( w 0 ))
(red dots); however, within 5 effective passes, the moving average (magenta dash) and upper-bound (blue line) of the step-size start surpassing the red dots, and eventually stablize above the conservative step-size.
For classical SARAH , we conﬁgure the algorithm with different values of the ﬁxed step-size, i.e., { 2 − 2 , 2 − 1 , ..., 2 4 } , and notice that 2 5 leads to a divergence.
On the other hand, AI-SARAH starts with a small step-size, yet achieves a faster convergence per effective pass with an eventual (moving average) step-size larger than 2 5 .
A LGORITHM
We present AI-SARAH in Algorithm 1.
This algorithm implicitly computes α t − 1 at any t ≥ 1 through approximately solving the sub-problem, i.e., min α>
ξ t ( α ) , and estimating the local Lipschitz smoothness of a stochastic function; the upper-bound of step-size makes the algorithm stable, and it is updated with exponential smoothing on harmonic mean (of approximate solutions to the subproblems), which also keep tracks of the local Lipschitz smoothness of a ﬁnite sum function, i.e.,
P ( w ) .
This algorithm is fully adaptive and requires no efforts of tuning, and can be implemented easily .
Notice that β is treated as a smoothing factor in updating the upper-bound of the step-size, and the default setting is β = 0 . 999 .
There exists one hyper-parameter in Algorithm 1, γ , which deﬁnes the early stopping criterion on Line 8, and the default setting is γ = 132 .
We will show later in this chapter that, the performance of this algorithm is not sensitive to the choices of γ , and this is true regardless of the problems (i.e., regularized/non-regularized logistic regression and different datasets.)
Algorithm
1: Parameter: 0 < γ < 1 (default 1 32 ), β = 0 .2:
Initialize: ˜
w
3: Set: α max = ∞ 4: for k
do 5: w
6: v
P
8: while (cid:107) v t
2 ≥ γ (cid:107) v
2 do 9:
Select random mini-batch S t from [ n ] uniformly with | S t | =
b 10: ˜ α t − 1 ≈ arg min α> 0 ξ t ( α ) 11: if k = 0 and t = 1 then 12: δ kt = 1 ˜
α t − 1 13: else 14: δ kt = βδ kt − 1 + (1 − β ) 1 ˜ α t −15: end if 16: α max = 1 δ kt 17: α t − 1 = min { ˜ α t − 1 , α max } 18: w t = w t − 1 − α t − 1 v t − 1 19: v t = ∇ f S t ( w t ) − ∇ f S t ( w t − 1 ) + v t −20: t = t +21: end while 22:
Set ˜ w k = w t .
23: end for
3. E STIMATE L OCAL L IPSCHITZ S MOOTHNESS
In the previous chapter, we showed that AI-SARAH adapts to local Lipschitz smoothness and yields a faster convergence than classical SARAH .
Then, the question is how to estimate the parameter of local Lipschitz smoothness in practice.
Can we use line-search?
The standard approach to estimate local Lipschitz smoothness is to use backtracking line-search.
Recall SARAH ’s update rule, i.e., w t = w t − 1 − α t − 1 v t − 1 , where v t − 1 is a stochastic recursive gradient.
The standard procedure is to apply line-search on function f i t ( w t − 1 − αv t − 1 ) .
However, the main issue is that − v t − 1 is not necessarily a descent direction.
AI-SARAH sub-problem.
Deﬁne the sub-problem (as shown on line 10 of Algorithm 1 3 ) as
[Equation] [Equation]
where t ≥ 1 denotes an inner iteration and i t indexes a random sample selected at t .
We argue that, by (approximately) solving (4), we can have good estimate of the parameters of the local Lipschitz smoothness.
To illustrate this setting, we denote L it the parameter of local Lipschitz smoothness prescribed by f i t at w t − 1 .
Let us focus on a simple quadratic function f
i t ( w ) = 12 ( x Ti t w − y i t ) 2 .
Let ˜ α be the optimal step-size along direction − v t − 1 , i.e. ˜ α = arg min α f i t ( w t − 1 − αv t − 1 ) .
Then, the closed form solution of ˜ α can be easily derived as ˜ α
= x Tit w t − 1 − y it x Tit v t −, whose value can be positive, negative, bounded or unbounded.
On the other hand, one can compute the step-size implicitly by solving (4) and obtain α it − 1 , i.e., α it − 1 = arg min
α ξ t ( α ) .
Then, we have
[Equation]
which is exactly 1 L it and recall L it is the parameter of local Lipschitz smoothness of f i t .
Simply put, as quadratic function has a constant Hessian, solving (4) gives exactly 1 L it .
For general (strongly) convex functions, if ∇ 2 f i t ( w t − 1 ) , does not change too much locally, we can still have a good estimate of L it by solving (4) approximately .
Based on a good estimate of L it , we can then obtain the estimate of the local Lipschitz smoothness of
P ( w t − 1 ) .
And, that is
[Equation]
Clearly, if a step-size in the algorithm is selected as 1 / ¯ L t , then a harmonic mean of the sequence of the step-size’s, computed for various component functions could serve as a good adaptive upper-bound on the step-size computed in the algorithm.
More details of intuition for the adaptive upper-bound can be found in Appendix A.2.
3. C OMPUTE S TEP - SIZE AND U PPER - BOUND On
Line 10 of Algorithm 1, the sub-problem is a one-dimensional minimization problem, which can be approximately solved by Newton method.
Speciﬁcally in Algorithm 1, we compute one-step
Newton at α = 0 , and that is
[Equation]
Note that, for convex function in general, (5) gives an approximate solution; for functions in particular forms such as quadratic ones, (5) gives an exact solution.
The procedure prescribed in (5) can be implemented very efﬁciently, and it does not require any extra (stochastic) gradient computations if compared with classical SARAH .
The only extra cost per iteration is to perform two backward passes, i.e., one pass for ξ (cid:48) t (0) and the other for ξ (cid:48)(cid:48) t (0) ; see
Appendix A.2 for implementation details.
As shown on Lines 11-16 of Algorithm 1, α max is updated at every inner iteration.
Speciﬁcally, the algorithm starts without an upper bound (i.e., α max = ∞ on Line 3); as ˜ α t − 1 being computed at every t ≥ 1 , we employs the exponential smoothing on the harmonic mean of { ˜ α t − 1 } to update the upper-bound.
For k ≥ 0 and t ≥ 1 , we deﬁne α max = 1 δ kt , where
[Equation]
otherwise and 0 < β < 1 .
We default β = 0 .
999 in Algorithm 1; see Appendix A.2 for details on the design of the adaptive upper-bound.
3. C HOICE OF γ
We perform a sensitivity analysis on different choices of γ .
Figures 2 shows the evolution of the squared norm of full gradient, i.e., (cid:107)∇ P ( w ) [Equation] 2 , for logistic regression on binary classiﬁcation problems; see extended results in Appendix A.
It is clear that the performance of γ ’s, where, γ ∈ { 1 / 8 , 1 / 16 , 1 / 32 , 1 / 64 } , is consistent with only marginal improvement by using a smaller value.
We default γ = 1 / 32 in Algorithm 1.
[Figure]
3. C ONVERGENCE A NALYSIS
In this section
[Figure]
we provide a convergence analysis of AI-SARAH (Algorithm 1) with a i) modiﬁed line 10 to ˜ α t − 1 ≈ arg min α ∈ [ α k min ,α k max ] ξ t ( α )
[Figure]
and ii) replacing the while loop with a for loop t ∈ [ m ] , where α k min and α k max are step-size bounds picked in each outer iteration and m be a hyper-parameter.
For brevity, let us just hint that for many problems (e.g., the one mentioned in Chapter 2), as the solution w t is approaching w ∗ the local curvature of P ( w ) is getting ﬂatter and hence α k max could be chosen as a fraction of reciprocal of a smoothness parameter of f i ( w ) ’s over set
[Equation]
which can be much larger than the fraction of reciprocal of a global smoothness parameter.
Let us just remark that for k -th outer loop, all iterates w t will stay inside of W k and hence α k max is well deﬁned.
The parameter 0 < α k min ≤ α k max can be chosen arbitrary (e.g., α k min =
α k max / 2 or even α k min =
α k max ).
We defer the technical details and proofs in Appendix B and C. Let us just state the main convergence theorem here.
Theorem 3.1.
Suppose that the functions f i ( w ) are convex and smooth with parameter L max k over
W k and P is µ -strongly convex.
Let us deﬁne
[Equation]
and select m and α k max such that σ km < 1 , ∀ k ≥ 1 .
Then, modiﬁed Algorithm 1 converges as follows:
[Equation]
Remark:
SARAH algorithm is a special case of the modiﬁed Algorithm 1 when ∀ k : α k min =
α k max = α ≤ 12 L .
N UMERICAL E XPERIMENT
In this chapter, we present the empirical study on the performance of AI-SARAH .
For brevity, we present a subset of experiments in the main paper, and defer the full experimental results and implementation details 4 in Appendix A.
The problems we consider in the experiment are (cid:96) 2 -regularized logistic regression for binary classiﬁcation problems; see Appendix A for non-regularized case.
Given a training sample ( x i , y i ) indexed by i ∈ [ n ] , the component function f
i is in the form f i ( w )
=
log(1 + exp( − y i
x Ti w ))
+ λ 2 (cid:107) w (cid:107) 2 ,
[Table]
where λ = 1 n for the (cid:96) 2 -regularized case and λ = 0 for the non-regularized case.
The datasets chosen for the experiments are ijcnn1, rcv1, real-sim, news20 and covtype .
Table 1 shows the basic statistics of the datasets.
More details and additional datasets can be found in Appendix A.
We compare AI-SARAH with SARAH , SARAH +, SVRG (Johnson & Zhang, 2013), ADAM (Kingma & Ba, 2015) and SGD with Momentum (Sutskever et al., 2013; Loizou & Richtárik, 2020; 2017).
While AI-SARAH does not require hyper-parameter tuning, we ﬁne-tune each of the other algorithms, which yields ≈ 5 , 000 runs in total for each dataset and case.
To be speciﬁc, we perform an extensive search on hyper-parameters: (1) ADAM and SGD with
Momentum ( SGD w/m) are tuned with different values of the (initial) step-size and schedules to reduce the step-size; (2) SARAH and SVRG are tuned with different values of the (constant) step-size and inner loop size; (3) SARAH + is tuned with different values of the (constant) step-size and early stopping parameter.
(See Appendix A for detailed tuning plan and the selected hyper-parameters.)
Figure 3 shows the average and total wall clock running time of AI-SARAH and the other algorithms.
While any individual run of AI-SARAH could be 2 - 5 x more time consuming than the other algorithms, its running time is negligible if comparing the total wall clock time.
The reason is that AI-SARAH does not require any tuning effort, but we have ≈ 5 , 000 runs to ﬁne-tune the other algorithms.
Figure 4 shows the minimum (cid:107)∇ P ( w ) (cid:107) 2 achieved at a few points of effective passes and wall clock time horizon.
It is clear that, AI-SARAH ’s practical speed of convergence is faster than the other algorithms in most cases.
Here, we argue that, if given an optimal implementation of AI-SARAH (just as that of
ADAM and other built-in optimizer in Pytorch 5 ), it is likely that our algorithm can be accelerated.
By selecting the ﬁne-tuned hyper-parameters of all other algorithms, we compare them with AISARAH and show the results in Figures 5-7.
For these experiments, we use 10 distinct random seeds to initialize w and generate stochastic mini-batches.
And, we use the marked dashes to represent the average and ﬁlled areas for 97% conﬁdence intervals.
Figure 5 presents the evolution of (cid:107)∇ P ( w ) (cid:107) 2 .
Obviously from the ﬁgure, AI-SARAH exhibits the strongest performance in terms of converging to a stationary point: by effective pass, the consistently large gaps are displayed between AI-SARAH and the rest; by wall clock time, we notice that
AI-SARAH achieves the smallest (cid:107)∇ P ( w ) (cid:107) 2 at the same time point.
This validates our design, that is to leverage local Lipschitz smoothness and achieve a faster convergence than SARAH and SARAH +.
In terms of minimizing the ﬁnite-sum functions, Figure 6 shows that, by effective pass, AI-SARAH consistently outperforms SARAH and SARAH + on all of the datasets with a possible exception on covtype dataset.
By wall clock time, AI-SARAH yields a competitive performance on all of the datasets, and it delivers a stronger performance on ijcnn1 and real-sim than SARAH .
For completeness of illustration on the performance, we show the testing accuracy in Figure 7.
Clearly, ﬁne-tuned ADAM dominates the competition.
However, AI-SARAH outperforms the other variance reduced methods on most of the datasets from both effective pass and wall clock time perspectives, and achieves the similar levels of accuracy as ADAM does on rcv1 , real-sim and covtype datasets.
Having illustrated the strong performance of AI-SARAH , we continue the presentation by showing the trajectories of the adaptive step-size and upper-bound in Figure 8.
This ﬁgure clearly shows that why AI-SARAH can achieve such a strong performance, especially on the convergence to a stationary point.
As mentioned in previous chapters, the adaptivity is driven by the local Lipschitz smoothness.
As shown in Figure 8, AI-SARAH starts with conservative step-size and upper-bound, both of which continue to increase while the algorithm progresses towards a stationary point.
After a few effective passes, we observe: the step-size and upper-bound are stablized
[Figure]
l
[Figure]
ll
[Figure]
l
[Figure]
c
[Figure]
s
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
i z e
[Figure] [Figure]
[Figure]
S t e p - S i z e [Figure]
due to λ (and hence strong convexity).
In Appendix A, we can see that, as a result of the function being unregularized, the step-size and upper-bound could be continuously increasing due to the fact that the function is likely non-strongly convex.
C ONCLUSION
In this paper, we propose AI-SARAH , a practical variant of stochastic recursive gradient methods.
The idea of design is simple yet powerful: by taking advantage of local Lipschitz smoothness, the step-size can be dynamically determined.
With intuitive illustration and implementation details, we show how AI-SARAH can efﬁciently estimate local Lipschitz smoothness and how it can be easily implemented in practice.
Our algorithm is tune-free and adaptive at full scale.
With extensive numerical experiment, we demonstrate that, without (tuning) any hyper-parameters, it delivers a competitive performance compared with SARAH(+) , ADAM and other ﬁrst-order methods, all equipped with ﬁne-tuned hyper-parameters.
E THICS S TATEMENT
This work presents a new algorithm for training machine learning models.
We do not foresee any ethical concerns.
All datasets used in this work are from the public domain and are commonly used benchmarks in ML papers.
R EPRODUCIBILITY S TATEMENT
We uploaded all the codes used to make all the experiments presented in this paper.
We have used random seeds to ensure that one can start optimizing the ML models from the same initial starting point as was used in the experiments.
We have used only datasets that are in the public domain, and one can download them from the following website https://www.csie.ntu.edu.tw/ ~cjlin/libsvmtools/datasets/ .
After acceptance, we will include a link to the GitHub repository where we will host the source codes.
A PPENDIX
The Appendix is organized as follows.
In Chapter A, we present extended details on the design, implementation and results of our numerical experiments.
In Chapter B, we present the theoretical analysis of AI-SARAH .
In Chapter C, we provide the basic deﬁnitions, some existing technical preliminaries that are used in our results, and the proofs of the main lemmas and theorems from
Chapter B.
A E XTENDED DETAILS ON N UMERICAL E XPERIMENT
In this chapter
[Table]
we present the extended details of the design, implementation and results of the numerical experiments.
A. P ROBLEM AND D ATA
The machine learning tasks studied in the experiment are binary classiﬁcation problems.
As a common practice in the empirical research of optimization algorithms, the LIBSVM datasets 6 are chosen to deﬁne the tasks.
Speciﬁcally, we selected 10 popular binary class datasets: ijcnn1, rcv1, news20, covtype, real-sim, a1a, gisette, w1a, w8a and mushrooms (see Table 2 for basic statistics of the datasets).
[Table] [Table]
A.1. D ATA P RE -P ROCESSING
Let ( χ i , y i ) be a training (or testing) sample indexed by i ∈ [ n ] (or i ∈
[ n test ] ), where χ
i ∈ R d −is a feature vector and y i is a label.
We pre-processed the data such that χ
i is of a unit length in
Euclidean norm and y i ∈ {− 1 , +1 } .
A.1. M ODEL AND L OSS F UNCTION
The selected model, h i : R d (cid:55)→ R , is in the linear form
[Equation] [Equation]
where ω ∈ R
d − 1 is a weight vector and ε ∈ R is a bias term.
For simplicity of notation, from now on, we let x i def =
[ χ Ti 1] T ∈ R
d be an augmented feature vector, w def
=
[ ω T ε ] T ∈ R
d be a parameter vector, and h i ( w ) =
x Ti w for i ∈ [ n ] .
Given a training sample indexed by i ∈ [ n ] , the loss function is deﬁned as a logistic regression
[Equation]
In (8), λ 2 (cid:107) w (cid:107) 2 is the (cid:96) 2 -regularization of a particular choice of λ > 0 , where we used λ = 1 n in the experiment; for the non-regularized case, λ was set to 0 .
Accordingly, the ﬁnite-sum minimization problem we aimed to solve is deﬁned as
[Equation]
Note that (9) is a convex function.
For the (cid:96) 2 -regularized case, i.e., λ = 1 /n in (8), (9) is µ -strongly convex and µ = 1 n .
However, without the λ , i.e., λ = 0 in (8), (9) is µ -strongly convex if and only if there there exists µ > 0 such that ∇ 2 P ( w ) (cid:23) µI for w ∈ R d (provided ∇ P ( w ) ∈ C ).
A. A LGORITHMS
This section provides the implementation details 7 of the algorithms, practical consideration, and discussions.
A.2. T UNE - FREE AI-SARAH
In Chapter 3 of the main paper, we introduced AI-SARAH , a tune-free and fully adaptive algorithm.
The implementation of Algorithm 1 was quite straightforward, and we highlight the implementation of Line 10 with details : for logistic regression, the one-dimensional (constrained optimization)
sub-problem min α> 0 ξ t ( α ) can be approximately solved by computing the Newton step at α = 0 , i.e., ˜
α t − 1 = −
[Equation]
[Equation]
(0) | .
This can be easily implemented with automatic differentiation in
Pytorch 8 , and only two additional backward passes w.r.t α is needed.
For function in some particular form, such as a linear least square loss function, an exact solution in closed form can be easily derived.
As mentioned in Chapter 3, we have an adaptive upper-bound, i.e., α max , in the algorithm. To be speciﬁc, the algorithm starts without an upper-bound, i.e., α max = ∞ on Line 3 of Algorithm 1. Then, α max is updated per (inner) iteration. Recall in Chapter 3, α max is computed as a harmonic mean of the sequence, i.e., { ˜ α t − 1 } , and an exponential smoothing is applied on top of the simple harmonic mean.
Having an upper-bound stabilizes the algorithm from stochastic optimization perspective .
For example, when the training error of the randomly selected mini-batch at w t is drastically reduced or approaching zero, the one-step Newton solution in (5) could be very large, i.e. ˜ α t − 1 (cid:29) 0 , which could be too aggressive to other mini-batch and hence Problem (1) prescribed by the batch.
On the other hand, making the upper-bound adaptive allows the algorithm to adapt to the local geometry and avoid restrictions on using a large step-size when the algorithm tries to make aggressive progress with respect to Problem (1).
With the adaptive upper-bound being derived by an exponential smoothing of the harmonic mean, the step-size is determined by emphasizing the current estimate of local geometry while taking into account the history of the estimates .
The exponential smoothing further stabilizes the algorithm by balancing the trade-off of being locally focused (with respect to f S t ) and globally focused (with respect to P ).
It is worthwhile to mention that Algorithm 1 does not require computing extra gradient of f S t with respect to w if compared with SARAH and SARAH+ .
At each inner iteration, t ≥ 1 ,
Algorithm 1 computes ∇ f S t ( w t − 1 − αv t − 1 ) with α = 0 just as SARAH and SARAH+ would compute ∇ f S t ( w t − 1 ) , and the only difference is that α is speciﬁed as a variable in Pytorch.
After the adaptive step-size α t − 1 is determined (Line 17), Algorithm 1 computes ∇ f S t ( w t − 1 − α t − 1 v t − 1 ) just as SARAH and SARAH+ would compute ∇ f
S t ( w t ) .
In Chapter 3 of the main paper, we discussed the sensitivity of Algorithm 1 on the choice of γ .
Here, we present the full results (on 10 chosen datasets for both (cid:96) 2 -regularized and non-regularized [Figure] [Figure] cases) in Figures 9, 10, 11, and 12.
Note that, in this experiment, we chose γ ∈ { 164 , 132 , 116 , 18 , 14 , 12 } , and for each γ , dataset and case, we used 10 distinct random seeds and ran each experiment foreffective passes.
A.2. O THER A LGORITHMS
In our numerical experiment, we compared the performance of TUNE-FREE AI-SARAH (Algorithm 1) with that of 5 FINE-TUNED state-of-the-art (stochastic variance reduced or adaptive) ﬁrst-order [Figure] [Figure] [Table]
[Table]
methods: SARAH , SARAH +, SVRG , ADAM and SGD with Momentum ( SGD w/m).
These algorithms were implemented in Pytorch, where ADAM and SGD w/m are built-in optimizers of Pytorch.
Hyper-parameter tuning.
For ADAM and SGD w/m, we selected 60 different values of the (initial) step-size on the interval [10 − 3 , 10] and 5 different schedules to decrease the step-size after every effective pass on the training samples; for SARAH and SVRG , we selected 10 different values of the (constant) step-size and 16 different values of the inner loop size; for SARAH +, the values of step-size were selected in the same way as that of SARAH and SVRG .
In addition, we chose 5 different values of the inner loop early stopping parameter.
Table 3 presents the detailed tuning plan for these algorithms.
Selection criteria:
We deﬁned the best hyper-parameters as the ones yielding the minimum ending value of the loss function, where the running budget is presented in Table 4.
Speciﬁcally, the criteria are: (1) ﬁltering out the ones exhibited a "spike" of the loss function, i.e., the initial value of the loss function is surpassed at any point within the budget; (2) selecting the ones achieved the minimum ending value of the loss function.
Hightlights of the hyper-parameter search:
• To take into account the randomness in the performance of these algorithms provided different hyper-parameters, we ran each conﬁguration with 5 distinct random seeds.
The total number of runs for each dataset and case is 4 , 850 .
• Tables 5 and 6 present the best hyper-parameters selected from the candidates for the regularized and non-regularized cases.
• Figures 13, 14, 15 and 16 show the performance of different hyper-parameters for all tuned algorithms; it is clearly that, the performance is highly dependent on the choices of hyperparameter for SARAH , SARAH +, and SVRG .
And, the performance of ADAM and SGD w/m are very SENSITIVE to the choices of hyper-parameter .
Global Lipschitz smoothness of P ( w ) .
Tuning the (constant) step-size of SARAH , SARAH + and
SVRG requires the parameter of (global) Lipschitz smoothness of P ( w ) , denoted the (global) Lipschitz
[Table]
[Table] [Figure]
[Table]
[Figure] [Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Figure] [Figure] [Figure] [Table]
constant L , and it can be computed as, given (8) and (9), [Equation] where λ max ( A ) denotes the largest eigenvalue of A and λ is the penalty term of the (cid:96) 2 -regularization in (8). Table 7 shows the values of L for the regularized and non-regularized cases on the chosen datasets.
[Figure] [Figure]
A. E XTENDED R ESULTS OF E XPERIMENT
In Chapter
[Figure]
we compared tune-free & fully adaptive AI-SARAH
(Algorithm 1) with ﬁne-tuned
SARAH , SARAH +, SVRG , ADAM and SGD w/m.
In this section, we present the extended results of our empirical study on the performance of AI-SARAH .
Figures 17 and 18 compare the average ending (cid:107)∇ P ( w ) (cid:107) 2 achieved by AI-SARAH with the other algorithms, conﬁgured with all candidate hyper-parameters.
It is clear that,
• without tuning, AI-SARAH achieves the best convergence (to a stationary point) in practice on most of the datasets for both cases; • while ﬁne-tuned ADAM achieves a better result for the non-regularized case on a1a, gisette, w1a and mushrooms , AI-SARAH outperforms ADAM for at least 80% ( a1a ), 55% ( gisette ), 50% ( w1a ), and 50% ( mushrooms ) of all candidate hyper-parameters.
Figure 19 shows the results of the non-regularized case for ijcnn1, rcv1, real-sim, news20 and covtype datasets.
Figures 20 and 21 present the results of the (cid:96) 2 -regularized case and non-regularized case respectively on a1a, gisette, w1a, w8a and mushrooms datasets.
For completeness of presentation, we present the evolution of AI-SARAH ’s step-size and upper-bound on a1a, gisette, w1a, w8a and mushrooms datasets in Figures 22 and 23.
Consistent with the results shown in Chapter 4 of the main paper, AI-SARAH delivers a competitive performance in practice.
[Figure] [Figure]
[Figure]
2 (middle row), and running maximum of testing accuracy (bottom row)
[Figure]
[Figure]
B T HEORETICAL A NALYSIS
In this chapter, we provide a convergence analysis of AI-SARAH (Algorithm 1) with a i) modiﬁed line 10 to ˜ α t − 1 ≈ arg min α ∈ [ α k min ,α k max ] ξ t ( α ) , and ii) replacing the while loop with a for loop t ∈ [ m ] , where α k min and α k max are step-size bounds picked in each outer iteration and m be a hyper-parameter.
B. AI-SARAH AND SARAH
Like SVRG and SARAH , AI-SARAH ’s loop structure of this algorithm is divided into the outer loop, where a full gradient is computed, and the inner loop, where only stochastic gradient is computed.
However, unlike SVRG and SARAH , the step-size is computed implicitly.
In particular, at each iteration t ∈ [ m ] of the inner loop, the step-size is chosen by approximately solving a simple onedimensional constrained optimization problem.
Deﬁne the (modiﬁed) sub-problem (optimization problem) at t ≥ 1 as
[Equation]
where ξ t ( α ) := (cid:107) v t (cid:107) 2 = (cid:107)∇ f S t ( w t − 1 − αv t − 1 ) −∇ f S t ( w t − 1 )+ v t − 1 (cid:107) 2 , α k min and α k max are lowerbound and upper-bound of the step-size respectively.
These bounds do not allow large ﬂuctuations of the (adaptive) step-size.
We denote α t − 1 the approximate solution of (10).
Now, let us present some remarks regarding AI-SARAH .
Remark B.1.
As we will explain with more details in the following subsections, the values of α k min and α k max cannot be arbitrarily large.
To guarantee convergence, we will need to assume that α k max ≤
[Equation]
k , where L max k = max i ∈ [ n ] L ik .
Here, L ik is the local smoothness parameter of f i deﬁned on a working-set for each outer loop (see Deﬁnition B.6).
Remark B.2.
SARAH Nguyen et al.
(2017) can be seen as a special case of AI-SARAH, where α k min =
α k max = α for all outer loops ( k ≥ 1 ).
In other words, a constant step-size is chosen for the algorithm.
However, if α k min < α k max , then the selection of the step-size in AI-SARAH allows a faster convergence of (cid:107) v t (cid:107) 2 than SARAH in each inner loop.
Remark B.3.
At t ≥ 1 , let us select a mini-batch of size n , i.e., | S t | = n .
In this case, AI-SARAH is equivalent to deterministic gradient descent with a very particular way of selecting the step-size, i.e. by solving the following problem
[Equation]
where ξ t ( α ) =
(cid:107)∇ P ( w t − 1 − α ∇ P ( w t − 1 ))
(cid:107) 2 .
In other words, the step-size is selected to minimize the squared norm of the full gradient with respect to w t .
B. D EFINITIONS / A SSUMPTIONS
First, we present the main deﬁnitions and assumptions that are used in our convergence analysis.
Deﬁnition B.4.
Function f : R d → R is L -smooth if: f ( x ) ≤ f ( y )
+ (cid:104)∇ f ( y ) , x − y
[Equation]
+ L 2 (cid:107) x − y (cid:107) 2 , ∀ x, y ∈ R d , and it is L C -smooth if:
[Equation]
Deﬁnition B.5.
Function f : R d → R is µ -strongly convex if: f ( x ) ≥ f ( y )
+ (cid:104)∇ f ( y ) , x − y (cid:105) + µ 2 (cid:107) x − y (cid:107) 2 , ∀ x, y ∈ R d .
If µ = 0 then function f is a (non-strongly) convex function.
Having presented the two main deﬁnitions for the class of problems that we are interested in, let us now present the working-set W k which contains all iterates produced in the k -th outer loop of (the modiﬁed) Algorithm 1.
Deﬁnition B.6 (Working-Set W k ) .
For any outer loop k ≥ 1 in (the modiﬁed) Algorithm 1, starting at ˜ w k − 1 we deﬁne
[Equation]
Note that the working-set W k can be seen as a ball of all vectors w ’s, which are not further away from ˜ w k − 1 than m · α k max (cid:107) v 0 (cid:107) .
Here, recall that m is the total number of iterations of an inner loop, α k max is an upper bound of the step-size α t − 1 , ∀ t ∈ [ m ] , and (cid:107) v 0 (cid:107) is simply the norm of the full gradient evaluated at the starting point ˜ w k − 1 in the outer loop.
By combining Deﬁnition B.4 with the working-set W k , we are now ready to provide the main assumption used in our analysis.
Assumption 1.
Functions
f
i , i ∈ [ n ] , of problem (1) are L i W k -smooth.
Since we only focus on the working-set W k , we simply write L ik -smooth.
Let us denote L i the smoothness parameter of function f
i , i ∈ [ n ] , in the domain R d .
Then, it is easy to see that L ik ≤ L i , ∀ i ∈ [ n ] .
In addition, under Assumption 1, it holds that function P is ¯
L k -smooth in the working-set W k , where ¯ L k = 1 n (cid:80) ni =1 L ik .
As we will explain with more details in the next section for our theoretical results, we will assume that α k max ≤ 2 L max k , where L max k = max i ∈ [ n ] L ik .
B.
C ONVERGENCE G UARANTEES
Now, we can derive the convergence rate of AI-SARAH .
Here, we highlight that, all of our theoretical results can be applied to SARAH .
We also note that, some quantities involved in our results, such as
L k and L max k , are dependent upon the working set W k (deﬁned for each outer loop k ≥ 1 ).
Similar to Nguyen et al.
(2017), we start by presenting two important lemmas, serving as the foundation of our theory.
The ﬁrst lemma provides an upper bound on the quantity (cid:80) mt =
E [ (cid:107)∇ P ( w t ) (cid:107) 2 ] .
Note that it does not require any convexity assumption.
Lemma B.7.
Fix a outer loop k ≥ 1 and consider Algorithm 1 with α k max ≤ 1 / ¯ L k .
Under
Assumption 1, (cid:80) mt
=
E [ (cid:107)∇ P ( w t ) (cid:107) 2 ] ≤
[Equation]
The second lemma provides an informative bound on the quantity E
[ (cid:107)∇ P ( w t ) − v t (cid:107) 2 ] .
Note that it requires convexity of component functions
f i , i ∈ [ n ] .
Lemma B.8.
Fix a outer loop k ≥ 1 and consider Algorithm 1 with α k max < 2 /L max k .
Suppose f i is convex for all i ∈ [ n ] .
Then, under Assumption 1, for any t ≥ 1 :
[Equation]
Equipped with the above lemmas, we can then present our main theorem and show the linear convergence of (the modiﬁed) Algorithm 1 for solving strongly convex smooth problems.
Theorem B.9.
Suppose that Assumption 1 holds and P is strongly convex with convex component functions
f i , i ∈ [ n ] .
Let us deﬁne
[Equation]
and select m and α k max such that σ km < 1 , ∀ k ≥ 1 .
Then, Algorithm 1 converges as follows:
[Equation]
As a corollary of our main theorem, it is easy to see that we can also obtain the convergence of
SARAH Nguyen et al.
Recall, from Remark B.2, that SARAH can be seen as a special case of
AI-SARAH if, for all outer loops, α k min =
α k max = α .
In this case, we can have
[Equation]
If we further assume that all functions
f
i , i ∈ [ n ] , are L -smooth and do not take advantage of the local smoothness (in other words, do not use the working-set W k ), then L max k =
L for all k ≥ 1 .
Then, with these restrictions, we have
[Equation]
As a result, Theorem B.9 guarantees the following linear convergence: E [ (cid:107)∇ P ( ˜ w k ) (cid:107) 2 ] ≤ ( σ m ) k
(cid:107)∇ P ( ˜ w 0 ) (cid:107) 2 , which is exactly the convergence of classical SARAH provided in Nguyen et al.
C T ECHNICAL P RELIMINARIES & P ROOFS OF M AIN R ESULTS
Let us start by presenting some important technical lemmas that will be later used for our main proofs.
C. T ECHNICAL P RELIMINARIES
Lemma C.1.
Nesterov (2003) Suppose that function f is convex and L -Smooth in C ⊆ R n .
Then for any w , w (cid:48) ∈ C :
[Equation] [Equation]
Lemma C.2.
Let Assumption 1 hold for all functions
f i of problem (1).
That is, let us assume that function f
i is L ik -smooth ∀ i ∈
[ n ] .
Then, function P ( w ) def = 1 n (cid:80) ni
=
f i ( w ) is ¯ L k -smooth, where ¯ L k = 1 n (cid:80) ni
=1 L ik .
Proof.
For each function f i , we have by deﬁnition of L ik -local smoothness,
[Equation]
Summing through all i (cid:48) s and dividing by n , we get
[Equation]
The next Lemma was ﬁrst proposed in Nguyen et al.
We add it here with its proof for completeness and will use it later for our main theoretical result.
Lemma C.3.
Nguyen et al.
(2017) Consider v t deﬁned in (2).
Then for any t ≥ 1 in Algorithm 1, it holds that:
[Equation]
Proof.
Let E j denote the expectation by conditioning on the information w
[Equation]
w 1 , . . . , w j as well as v 0 , v 1 , . .
. , v j − 1 .
Then,
[Equation] [Equation]
where the last equality follows from
[Equation]
By taking expectation in the above expression, using the tower property
[Equation]
and summing over j = 1 , ..., t , we obtain
[Equation]
C. P ROOFS OF L EMMAS AND T HEOREMS
For simplicity of notation, we use | S | = 1 in the following proofs, and a generalization to | S | > 1 is straightforward.
C.2. P ROOF OF L EMMA B.
By Assumption
[Equation]
Lemma C
[Equation]
2 and the update rule w t = w t − 1 − α t − 1 v t − 1 of Algorithm
[Equation]
we obtain:
[Equation]
where
[Equation]
in the equality above
[Equation]
we use the fact that (cid:104)
a
[Equation]
By rearranging and using the lower and upper bounds of the step-size α t − 1 in the outer loop k ( α k min ≤ α t − 1 ≤ α k max )
[Equation]
we get: α k min
[Equation]
By assuming that α k max ≤ 1¯ L k , it holds that α t − 1 ≤ 1¯ L k and (cid:0) 1 − ¯ L k α t − 1 (cid:1) ≥ 0 , ∀ t ∈ [ m ] .
Thus, α k min
[Equation]
By taking expectations and multiplying both sides with
[Equation]
where the last inequality holds as α k max ≤ 1¯ L k .
Summing over t = 1 , 2 , . .
, m + 1 , we have m +1 (cid:88)
[Equation]
where the last inequality holds since w ∗ is the global minimizer of P
[Equation]
The last expression can be equivalently written as:
[Equation]
which completes the proof.
C.2. P ROOF OF L EMMA B.
[Equation]
For each outer loop k , it holds that α j − 1 ≤ α k max and L ik ≤ L max k .
Thus,
[Equation]
By rearranging
[Equation]
taking expectations again
[Equation]
and assuming that α k max < 2 /L max k
[Equation]
By summing the above inequality over j = 1 , . . .
, t ( t ≥ 1) , we have:
[Equation]
Now, by using Lemma C
[Equation] [Equation]
we obtain:
[Equation]
C.2. P ROOF OF T HEOREM B.
Proof.
Since v 0 = ∇ P ( w 0 ) implies (cid:107)∇ P ( w 0 ) − v 0 (cid:107) 2 = 0 , then by Lemma B
[Equation]
8, we obtain: m
[Equation] [Equation]
Combine this with Lemma B.7, we have that:
[Equation]
Since we are considering one outer iteration, with k ≥
[Equation]
we have v 0 = ∇ P ( w 0 ) =
∇ P ( ˜ w k − 1 ) and ˜ w k = w t , where t is drawn uniformly at random from { 0 , 1 , . .
, m } .
Therefore, the following holds,
E [
[Equation]
Let us use σ km =
[Equation]
, then the above expression can be written as:
[Equation]
By expanding the recurrence
[Equation]
we obtain:
[Equation]
This completes the proof.