S TOCHASTIC R EWEIGHTED G RADIENT D ESCENT
Anonymous authors
Paper under double-blind review
A BSTRACT
Importance sampling is a promising strategy for improving the convergence rate of stochastic gradient methods.
It is typically used to precondition the optimization problem, but it can also be used to reduce the variance of the gradient estimator.
Unfortunately, this latter point of view has yet to lead to practical methods that provably improve the asymptotic error of stochastic gradient methods.
In this work, we propose stochastic reweighted gradient (SRG), a variance-reduced stochastic gradient method based solely on importance sampling that can improve on the asymptotic error of stochastic gradient descent (SGD) in the strongly convex and smooth case.
We show that SRG can be extended to combine the beneﬁts of both importance-sampling-based preconditioning and variance reduction.
When compared to SGD, the resulting algorithm can simultaneously reduce the condition number and the asymptotic error, both by up to a factor equal to the number of component functions.
We demonstrate improved convergence in practice on (cid:96) 2 -regularized logistic regression problems.
I NTRODUCTION
Unconstrained optimization of ﬁnite-sum objectives is a core algorithmic problem in machine learning.
The prototypical way of solving such problems is by viewing them through the lens of stochastic optimization, where the source of stochasticity resides in the choice of the index in the sum.
Stochastic gradient descent (SGD) (Robbins & Monro, 1951) remains the standard algorithm for this class of problems.
A natural way to improve on SGD is by considering importance sampling schemes.
This idea is not new and dates back to (Needell et al., 2014) who uses importance sampling as a preconditioning technique.
They propose sampling the indices with probabilities proportional to the smoothness constants of the corresponding component functions, and show that this sampling scheme provably reduces the condition number of the problem.
In another line of work, variance-reduced methods were found to achieve linear convergence in the strongly convex and smooth case (Roux et al., 2012; Schmidt et al., 2017).
Many of these methods rely on control variates to reduce the variance of the gradient estimator used by SGD (Johnson & Zhang, 2013; Defazio et al., 2014).
While very successful, the applicability of these methods is limited by the large memory overhead that they introduce, or the periodic full-gradient recomputation that they require (Defazio & Bottou, 2019).
Despite strong progress in this research area, an importance-sampling-based analogue to these algorithms, which is potentially free from these drawbacks, has yet to emerge.
In this work, we propose such an analogue.
We introduce stochastic reweighted gradient (SRG), an importance-sampling-based variance-reduced optimization algorithm.
Similar to SGD, SRG requires a single gradient oracle call per iteration, and only requires O ( n ) additional memory, and
O (log n ) additional ﬂoating point operations per iteration.
We analyze the convergence rate of SRG in the strongly-convex and smooth case, and show that it can provably improve the asymptotic error of SGD.
Finally, we show how our importance sampling strategy can be combined with smoothnessbased importance sampling, and prove that the resulting algorithm simultaneously performs variance reduction and preconditioning.
We demonstrate improved convergence in practice on (cid:96) 2 -regularized logistic regression problems.
P RELIMINARIES
We consider the ﬁnite-sum optimization problem:
[Equation]
where F is µ -strongly convex for µ > 0 , and for all i ∈ [ n ] , f i is convex and L i -smooth for
L i > 0 .
Note that by strong-convexity, F has a unique minimizer x ∗ ∈ R d .
We deﬁne the maximum
L max := max i ∈ [ n ] L i and average L :=
(cid:80) nj
=
L
i /n smoothness constants.
Similarly, we deﬁne the maximum κ max :=
L max /µ and average κ :=
L/µ condition numbers.
The classical way of solving (1) is by viewing it as a stochastic optimization problem where the randomness comes from the choice of the index i ∈ [ n ] .
Starting from some x 0 ∈ R d , and for an iteration number k ∈ N , stochastic gradient descent (SGD) performs the following update:
[Equation]
for a step size α k > 0 and a random index i k drawn uniformly from [ n ] .
The idea behind importance sampling for SGD is to instead sample the index
i k
according to a chosen distribution p k on [ n ] , and to perform the update (Needell et al., 2014):
[Equation]
where p ik is the i th component of the probability vector p k .
It is immediate to verify that the importance sampling estimator of the gradient is unbiased as long as p k > 0 .
The question that we address in this paper is how to design a sequence { p k } ∞ k =0 that produces more efﬁcient gradient estimators than the ones produced by uniform sampling.
One way to design such a sequence is by adopting a greedy strategy: at each iteration k we choose p k to minimize the conditional variance of the gradient estimator, which is given by, up to an additive constant:
[Equation]
This conditional variance is minimized at (Zhao & Zhang, 2015):
[Equation]
Ideally, we would like to set p k to this minimizer.
However, this requires knowledge of the gradient norms ( (cid:107)∇ f i ( x k ) (cid:107) 2 ) ni =1 , which, in general, requires n gradient evaluations per iteration.
A LGORITHM
In this section, we show how to design a tractable sequence of importance sampling distributions for SGD that approximate the conditional-variance-minimizing distributions (4).
First we construct efﬁcient approximations of the conditional variances (3).
We then state a simple bound on the approximation errors and use it to motivate our choice of importance sampling distributions.
To approximate the conditional variances, we follow the strategy of certain variance-reduced methods (Roux et al., 2012; Schmidt et al., 2017; Defazio et al., 2014).
These methods maintain a table ( g i k ) n i =1 that tracks the component gradients ( ∇ f i ( x k ))
n i =1 and updates itself each iteration at the index
i k used to update the iterates x k (2).
Our method instead maintains an array of gradient norms , from which we construct an approximation of the conditional variance (3) of the gradient estimator:
[Equation]
− k is minimized at
[Equation]
We do not directly use q k as an importance sampling distribution, because this approximation may be poor.
In particular, we have the following bound on the conditional variance (3):
[Equation]
Recall that our goal is to pick p k that minimizes σ 2 ( x k , p ) .
q k minimizes the second term on the right-hand side, but we must ensure that both terms are small.
Two conditions are needed to keep the ﬁrst term small.
The ﬁrst is to control the terms (cid:13)(cid:13) ∇ f i ( x k )
− g ik (cid:13)(cid:13) 2 2 , which we can do by making sure that the historical gradients g ik are frequently updated.
The second is to ensure that the probabilities p ik are lower bounded.
We achieve both of these properties, as well as approximately minimize the right-hand side of (6) by mixing q k with the uniform distribution over [ n ] .
This yields our ﬁnal importance sampling distribution for a given mixture coefﬁcient θ k ∈ (0 , 1] :
[Equation]
Our analysis clariﬁes the role of the sequence of mixture coefﬁcients ( θ k ) ∞ k =0 , and relates it to both the step size sequence ( α k ) ∞ k =0 and the asymptotic error of SRG in the constant step size setting.
Curiously, our analysis requires performing the update of the array ( (cid:13) (cid:13)
g i k
(cid:13) (cid:13) 2 ) ni
=1 only when the index i k is drawn from the uniform mixture component.
It is not clear to us whether this constraint is an artifact of the analysis or a property of the algorithm.
We discuss this further after the statement of
Lemma 1 in section 4.
The pseudocode for our method is given in Algorithm 1.
We brieﬂy discuss how SRG can be efﬁciently implemented.
To sample from q k , we store ( (cid:13)(cid:13)
g ik (cid:13)(cid:13) 2 ) ni
=1 in a binary indexed tree using O ( n ) memory, along with the normalizing constant λ k = (cid:80) ni
=1 (cid:13)(cid:13)
g ik (cid:13)(cid:13) 2 .
The binary indexed tree can be updated and maintained in O (log n ) operations, while the normalizing constant can be maintained in constant time.
We can then sample from q k using inverse transform sampling: we multiply a uniform random variable u ∈ [0 , 1) by λ k , then return the largest index whose corresponding preﬁx sum is less than λ k u .
This procedure also requires only O (log n ) operations.
The total overhead of SRG when compared to SGD is therefore
O ( n ) memory and O (log n ) ﬂoating point operations per iteration.
T HEORY
In this section, we analyze the convergence rate of SRG, and show that it can achieve a better asymptotic error than SGD.
Two key constants are helpful in contrasting the asymptotic errors of
[Equation]
It is well known that the asymptotic error of SGD depends linearly on σ 2 (Needell et al., 2014).
We here show that SRG reduces this to a linear dependence on σ 2 ∗ , which can be up to n times smaller.
To study the convergence rate of SRG, we use the following Lyapunov function, which is similar to the one used to study the convergence rate of SAGA (Hofmann et al., 2015; Defazio, 2016):
[Equation]
for a constant a > 0 that we set during the analysis.
The proofs of this section are in Appendix B.
I NTERMEDIATE LEMMAS
Before proceeding with the main result, let us ﬁrst state two intermediate lemmas.
The ﬁrst studies the evolution of ( g ik ) ni =1 from one iteration to the next.
Lemma 1.
Let k ∈ N and suppose that ( g ik ) ni
=1 evolves as in Algorithm 1.
Taking expectation with respect to ( b k , i k ) , conditional on ( b t , i t ) k − 1 t =0 , we have:
[Equation]
The use of the Bernoulli random variable b k in Algorithm 1 to monitor the update of ( g ik ) ni =1 is necessary for Lemma 1 to hold.
In particular, without the use of b k , the elements of ( g ik ) ni =1 may have different probabilities of being updated.
In that case, the second term of Lemma 1 becomes a weighted average of terms that are technically difﬁcult to deal with.
When the importance sampling distribution does not depend on the iteration, this issue can be ﬁxed with a slight modiﬁcation of the Lyapunov function (8) (Schmidt et al., 2015).
In our case however, we are dealing with timevarying importance sampling distributions, and this approach fails.
This is why we rely on the
Bernoulli random variable b k to ensure that the probability of updating any g ik is ﬁxed to θ k /n .
The second lemma is a bound on the conditional variance of the gradient estimator used by SRG.
We intentionally leave as many free parameters as possible in the bound and optimize over them in the main result of section 4.2.
Lemma 2.
Let k ∈ N and assume that θ k ∈ (0 , 1 / 2] .
Taking expectation with respect to ( b k , i k ) , conditional on ( b t , i t ) k − 1 t =0 , we have, for all β, γ, δ, η > 0 :
[Equation]

[Equation]
where D 1 , D 2 and D 3 are positive functions of the free parameters β, γ, δ, η .
4. M AIN RESULT
Our main result is a bound on the evolution of the Lyapunov function T k along the steps of SRG.
Theorem 1.
Suppose that ( x k , ( g ik ) ni =1 ) evolves according to Algorithm 1.
Further, assume that for all k ∈ N : (i) α k /θ k is non-increasing.
(ii) θ k ∈ (0 , 1 / 2] .
(iii) α k ≤ θ k / 12 L max .
Then:
[Equation]
for all k ∈ N , and where:
[Equation]
The constants ( 1 / 12 in the bound on the step size and in ρ k , 6 in front of the σ 2 ∗ term) in this theorem are optimized under the following constraints.
First, the parameterized form of the above bound shows that the largest allowable step size is given by c 2 θ k /L max .
Using it we get:
[Equation]
for some constants c 1 , c 2 > 0 .
As we do not know a priori the relative magnitudes of n and κ max , and since c 1 and c 2 are inversely proportional, we impose the constraint c
= c 2 .
Similarly, our parameterized bound gives an asymptotic error of the form (1 + 2 θ k ) c
3 α 2 k σ 2 ∗ for a constant c 3 .
We chose to impose the constraint c 3 = 6 .
c 1 , c 2 and c 3 are all functions of the free parameters of Lemma 2 and the constant a of the Lyapunov function (8).
Numerically maximizing c 2 (and therefore the largest allowable step size) subject to these two constraints ( c 3 = 6 and c 1 = c 2 ) with respect to these parameters yields the result in Theorem 1.
To give the reader an idea of the sensitivity of the result to the choice of c 3 , note that setting c 3 =yields c 2 ≈ 1 / 20 , whereas taking c 3 (cid:29) 1 yields c 2 ≈ 1 / 10 .
We have attempted to obtain the best bound possible on the largest step size allowable, but the rather small prefactor c 2 seems to be an inevitable consequence of the multiple (but as far as we can tell necessary) uses of Young’s inequality in our analysis.
Our experiments suggest that the dependence on the mixture coefﬁcient θ is real, but that the prefactor c 2 = 1 / 12 may be an artifact of the analysis.
For SRG with a constant mixture coefﬁcient and step size, its convergence rate and complexity are characterized by the following corollary of Theorem 1:
Corollary 1.
Suppose that ( x k , ( g ik ) ni =1 ) evolves according to Algorithm 1 with a constant mixture coefﬁcient θ k = θ ∈ (0 , 1 / 2] and a constant step size α k
= α ≤ θ/ 12 L max .
Then for any k ∈ N :
[Equation]
where ρ = ρ k is as deﬁned in Theorem 1.
For any ε > 0 and θ ∈ (0 , 1 / 2] , choosing:
[Equation]
and:
[Equation]
Comparing the convergence rate of SRG in Corollary 1 with the standard result for SGD (Needell et al., 2014), we see that they are of similar form.
When ρ = αµ , the bound of Corollary 1 is better asymptotically.
Indeed, in this case, and as k → ∞ , the iterates of SRG stay within a ball of radius
O ( (cid:112) σ 2 ∗ ) of the minimizer, while those of SGD stay withing a ball of radius O ( √ σ 2 ) .
The equality ρ = αµ holds when α ≤ 1 /nµ , which is true for all allowable step sizes if the problem is dominated by its maximum condition number κ max ≥ n , and for small step sizes otherwise.
In terms of complexity, we have the following comparison.
Up to constants, the complexity of SRG with a constant mixture coefﬁcient and step size is of the form:
[Equation]
We compare this to the complexity of SGD with constant step size (Needell et al., 2014):
[Equation]
In the high accuracy regime, the ε − 1 terms dominate the complexities of SRG and SGD.
In this case,
SRG enjoys a better complexity than SGD since σ 2 ∗ ≤ σ 2 .
Parameters: step sizes
∞
0 > 0 , mixture coefﬁcients
Initialization for k = 0 , 1 , 2 , . . .
do p
{ q is deﬁned in v is deﬁned in if otherwise end for
E XTENSION
In this section, we extend SRG to combine its variance reduction capacity with the preconditioning ability of smoothness-based importance sampling.
A straightforward way to generalize the argument given in the derivation of SRG in section 3 is to consider the following bound on the conditional variance (3) of the gradient estimator, which can be derived from Young’s inequality and the L i -smoothness of each f i :
[Equation]
While the motivating bound (6) seems more intuitive, because we think of g ik as tracking ∇ f i ( x k ) , it turns out that this second bound better captures the evolution of g ik in relation to ∇ f i ( x k ) .
At a high-level, this is because g ik tracks ∇ f i ( x k ) indirectly: both hover around ∇ f i ( x ∗ ) as k gets large.
Similar to our approach in section 3, our goal is to pick p k that minimizes the right-hand side of (9).
We know that q k (5) minimizes the third term, but we need to make sure that the ﬁrst two are also small.
To minimize the ﬁrst term, knowing nothing about the relative sizes of the inner product terms, it makes sense to have probabilities proportional to the smoothness constants 1 .
On the other hand, to keep the second term small, we need to ensure that the historical gradients ( g ik ) ni = are frequently updated, which we can do by imposing a uniform lower bound on the probabilities.
These considerations motivate us to consider the following distributions:
[Equation]
for positive mixture coefﬁcients ( θ k , η k ) satisfying θ k + η k ∈
[Equation]
0 , 1] , and where v is given by:
[Equation] [Equation]
Using these probabilities, we are able to show that the resulting algorithm does indeed achieve both variance reduction and preconditioning.
However, in the worst case, its complexity is twice as much as what we would expect from simply replacing L max with L in Corollary 1.
Intuitively, this is because the probability assigned to the uniform component in (7) needs to be split between the uniform and the smoothness-based components in (10).
5. C AREFULLY DECOUPLING THE UPDATES OF ( g ik ) ni =1 AND x k
Here we show how to design our algorithm such that this additional factor of 2 (described above) in the complexity is reduced to:
[Equation]
Following (Schmidt et al., 2015), our method is based on the observation that we can decouple the index used to update the historical gradients and the index used to update the iterates, which we refer to as j k and i k , respectively.
Intuitively, to minimize (10) we would ideally want j k to be uniformly distributed and i k to be distributed according to p k
= (1 − θ k ) q k
+ θ k v .
Unfortunately, sampling ( j k , i k ) independently with these marginals does not address our issue, because we would still require an average of approximately two gradient evaluations per iteration.
Luckily, we can use any coupling between ( j k , i k ) , because the evolution of the Lyapunov function (8) only depends on the marginal distributions and not the joint.
In particular, we obtain the same bound on the evolution of T k regardless of how i k and j k are coupled.
It therefore makes sense to pick the coupling that maximizes the probability that i k
= j k since this minimizes the number of gradient evaluations required per iteration.
Such couplings are known as maximal couplings in the literature (see, e.g., Biswas et al.
and can easily be computed for discrete random variables.
With a maximal coupling, the expected number of gradient evaluations per iteration becomes 1 + (cid:107) v − 1 /n (cid:107) TV .
Using this idea we arrive at SRG + described in Algorithm 2.
Let us brieﬂy discuss the implementation of SRG + .
Sampling from π , the maximal coupling of v and the uniform distribution, requires forming three probability vectors (see, e.g., Biswas et al.
As both v and 1 /n are constant throughout the optimization process, we can form these vectors at the beginning of the algorithm along with their partial sums for a total initial cost of O ( n ) operations.
We can then sample from π in O (log n ) time using binary search on the partial sums at each iteration.
Adding this sampling therefore does not change the overhead of SRG.
5. A NALYSIS
The analysis of SRG + is similar to that of SRG.
In particular, the iterates of SRG + also obey a slight modiﬁcation of Theorem 1 where the bound on the largest allowable step size is loosened to α k ≤ θ k / 12 L .
We refer the reader to Appendix C for more details.
Due to this improvement, we get that the complexity of SRG + with a constant mixture coefﬁcient and step size is given by:
[Equation]
This shows that SRG + performs both variance reduction as shown by the dependence of the complexity on σ 2 ∗ instead of σ 2 and preconditioning as shown by the dependence on κ instead of κ max .
R ELATED WORK
At a high-level, three lines of work exist that study the use of importance sampling with SGD.
The ﬁrst one considers ﬁxed importance sampling distributions based on the constants of the problem (Needell et al., 2014; Zhao & Zhang, 2015), and shows that such a strategy leads to improved conditioning of the problem.
The second considers adaptive importance sampling and similar to our work targets the variance of the gradient estimator, but generally fails at providing strong convergence rate guarantees under standard assumptions (Papa et al., 2015; Alain et al., 2016;
Canevet et al., 2016; Stich et al., 2017; Katharopoulos & Fleuret, 2018; Johnson & Guestrin, 2018).
The third line of work frames the problem as an online learning problem with bandit feedback and provides guarantees on the regret of the proposed distributions in terms of the variance of the resulting estimators (Namkoong et al., 2017; Salehi et al., 2017; Borsos et al., 2018; 2019; El Hanchi & Stephens, 2020).
Our method is closely related to the one proposed by (Papa et al., 2015); their analysis however requires non-standard assumptions and their main result is asymptotic in nature.
In contrast, our work is the ﬁrst to provide non-asymptotic guarantees on the suboptimality of the iterates for variance-reducing importance sampling under standard technical assumptions.
E XPERIMENTS
In this section, we empirically verify our two main claims: (i) SRG performs variance reduction which can improve the asymptotic error of SGD.
(ii) SRG + performs both variance reduction and preconditioning, and can both reduce the asymptotic error of SGD and allow the use of larger step sizes.
We with controlled synthetic experiments that provide direct support for our claims.
We then compare SRG to other baseline optimizers on
2 -regularized logistic regression problems.
In all experiments, we ran each algorithm 10 times and present the averaged result.
7. S YNTHETIC EXPERIMENTS
For our ﬁrst experiment, we consider the following toy problem.
We let x ∈ R and f i ( x ) = 12 ( x − a i )
where a i = 0 for i ∈ [ n − 1] and a n = 1 .
In this case, x ∗ = 1 /n , σ 2 ≈ 1 /n , and σ 2 ∗ ≈ 4 /n 2 .
We consider ﬁve instantiations of this problem with n ∈ { 8 , 16 , 32 , 64 , 128 } , yielding ratios σ 2 /σ 2 ∗ approximately in { 2 , 4 , 8 , 16 , 32 } .
For each instantiation, we ran SGD and SRG until they reached stationarity and recorded their asymptotic errors
lim k →∞ E (cid:104) (cid:107) x
k − x ∗ (cid:107) 22 (cid:105) , which we denote by ∆ SGD and ∆ SRG respectively.
We experimented with three different step sizes, two of them allowed by Corollary 1, and one larger one for which we can only prove that SRG has a similar convergence guarantee as SGD.
For SRG, we used the mixture coefﬁcient θ = 1 / 2 .
We plot ∆ SGD / ∆ SRG against σ 2 /σ 2 ∗ in Figure 1 (left) for each of the three step sizes, from which we see that the relationship between the two ratios is linear, and very close to identity.
From an asymptotic error point of view, these results support our theory in that the improvement is seen to be directly proportional to the ratio σ 2 /σ 2 ∗ .
On the other hand, the constant 6(1 + 2 θ ) /ρ in Corollarywould suggest that the proportionality constant is much smaller than 1, particularly when n is large, but this is not what we observe in practice.
This could be because the ﬁrst term of the Lyapunov function (8) is quite large at stationarity, or because the constants in our bound are not sharp due to the multiple uses of Young’s inequality.
This latter possibility is further supported by the fact that we see a similar behaviour for SRG for step sizes larger than the ones allowed by our theory.
We have consistently made these two observations in other experiments.
It is however unclear to us how our analysis can be improved to match these observations.
For our second experiment, we considered the following problem.
We let x ∈ R and f i ( x ) = 12 L i ( x − a i ) 2 , and ﬁxed n = 20 .
Similar to the ﬁrst experiment, we take a i = 0 for i ∈
[ n − 1] and a n = 1 .
We then set L 1 = n − 1 , L n = 1 /n , and L i = n − 1 n ( n − 2) , so that L = 1 , L max = n − 1 .
In this case, we get x ∗ = 1 /n 2 , σ 2 + = σ 2 ( x ∗ , v )
≈ 1 /n , σ
≈ 2 /n 2 , and σ 2 ∗ ≈ 4 /n 3 .
We then ran SGD with L i -sampling (SGD + ), SGD with partially biased sampling (SGD ++ , Needell et al.
(2014)), and
SRG + , all with the largest allowable step size α = θ/ 12 L from our theory for SRG + , and we use the mixture coefﬁcient θ = 1 / 2 for both SRG + and SGD ++ .
We have obtained very similar results when using the larger step size α = θ/ 2 L , which is the maximum allowable for SGD + .
We plot the relative error E (cid:104) (cid:107) x k − x ∗ (cid:107) 22 (cid:105) / (cid:107) x 0 − x ∗ (cid:107) 22 for each algorithm against the number of gradient oracle calls it makes in Figure 1 (right).
We see that all three algorithms are able to converge even when using the large O (1 /L ) step sizes.
The theory of SGD + allows for larger step sizes compared to SGD, but the asymptotic error of the algorithm depends on σ
+ (Needell et al., 2014), which can be up to n times bigger than σ 2 .
To solve this problem, Needell et al.
(2014) proposed partially biased sampling which mixes the smoothness-based distribution with uniform sampling, allowing the use of larger step sizes just like SGD+, but preserving the asymptotic error of SGD.
SRG + further improves on SGD++, and reduces the asymptotic error, making it proportional to σ 2 ∗
7. (cid:96) 2 - REGULARIZED LOGISTIC REGRESSION
For our last experiment, we test SRG on (cid:96) 2 -regularized logistic regression problems.
In this case, the functions f i are given by:
[Equation]
where y i ∈ { 0 , 1 } is the label of data point a i ∈ R d .
Each f i is convex and L i = 0 .
25 (cid:107) a i (cid:107) 22 + µ smooth.
Their average F is also µ -strongly convex.
As is standard, we select µ = 1 /n .
We experiment with SRG on four datasets from LIBSVM (Chang & Lin, 2011): ijcnn1, w8a, mushrooms and phishing .
For each dataset, and to be able to efﬁciently run our experiments, we randomly select a subset of the data of size n = 1000 .
As the datasets are normalized, we have that (cid:107) a i (cid:107) 2 =for all i ∈ [ n ] .
This makes L i =
L = 0 .
25 + µ , which reduces SGD + to SGD, and SRG + to SRG.
We tested the performance of SRG against three baselines, the ﬁrst of which is standard SGD.
The second is SGD with the optimal variance-minimizing probabilities p ik ∝ (cid:107)∇ f i ( x k ) (cid:107) 2 , which allows us to compare SRG with the best possible variance-reducing importance sampling scheme.
The last baseline is SGD with random shufﬂing, which is also known to improve the asymptotic error of SGD (under the additional assumption that each f i is also strongly convex), (Mishchenko et al., 2020).
We evaluate the performance of the algorithms by tracking the average relative error
E (cid:104) (cid:107) x k − x ∗ (cid:107) 22 (cid:105) / (cid:107) x 0 − x ∗ (cid:107) 22 .
We used the mixture coefﬁcient θ = 1 / 2 for SRG, and used the same step size α = θ/ 2 L for all algorithms.
The results of this experiment are shown in Figure 2.
We observe that SRG consistently outperforms SGD on all datasets, and that it closely matches the performance of SGD with the varianceminimizing distributions, which it tries to approximate.
We also see that SRG outperforms SGD with random shufﬂing on two datasets, and is competitive with it on the remaining two.
C ONCLUSION
We introduced SRG, a new importance-sampling based variance-reduced optimization algorithm for ﬁnite-sum problems.
We analyzed its convergence rate in the strongly convex and smooth case, and showed that it can improve on the asymptotic error of SGD.
We also introduced SRG + , an extension of SRG which simultaneously performs variance reduction and preconditioning through importance sampling.
We expect our algorithms to be most useful in the medium accuracy regime, where the required accuracy is higher than the one achieved by SGD, but low enough that the overhead of classical variance reduced methods becomes signiﬁcant.
Finally, an interesting future direction would be to explore non-greedy strategies for the design of importance sampling distributions for
SGD that not only minimize the variance of the current gradient estimator, but also take into account the variance of the gradient estimators of subsequent iterations.