Multi-block-Single-probe Variance Reduced Estimator for Coupled Compositional Optimization
Abstract
Variance reduction techniques such as SPIDER/SARAH/STORM have been extensively studied to improve the convergence rates of stochastic non-convex optimization, which usually maintain and update a sequence of estimators for a single function across iterations.
What if we need to track multiple functional mappings across iterations but only with access to stochastic samples of O (1) functional mappings at each iteration?
There is an important application in solving an emerging family of coupled compositional optimization problems in the form of (cid:80) mi =
f
i
g i ( w )) , where g i is accessible through a stochastic oracle.
The key issue is to track and estimate a sequence of g ( w )
=
( g 1 ( w ) , . . .
, g m ( w )) across iterations, where g ( w ) has m blocks and it is only allowed to probe O (1) blocks to attain their stochastic values and Jacobians.
To improve the complexity for solving these problems, we propose a novel stochastic method named Multi-block-Singleprobe Variance Reduced (MSVR) estimator to track the sequence of g ( w ) .
It is inspired by STORM but introduces a customized error correction term to alleviate the noise not only in stochastic samples for the selected blocks but also in those blocks that are not sampled.
With the help of the MSVR estimator, we develop several algorithms for solving the aforementioned compositional problems with improved complexities across a spectrum of settings with non-convex/convex/strongly convex/Polyak-Łojasiewicz (PL) objectives.
Our results improve upon prior ones in several aspects, including the order of sample complexities and dependence on the strong convexity parameter.
Empirical studies on multi-task deep AUC maximization demonstrate the better performance of using the new estimator
Introduction
This paper is motivated by solving the following Finite-sum Coupled Compositional Optimization (FCCO) problem that has broad applications in machine learning [Wang and Yang, 2022]:
[Equation]
where f i : R p (cid:55)→ R is a simple deterministic function.
We assume that only noisy estimations of g i ( · ) and its Jacobian ∇ g i ( · ) can be accessed, denoted as g i ( · ; ξ i ) and ∇ g i ( · ; ξ i ) , where ξ i represents the random sample(s) drawn from a stochastic oracle such that E [ g i ( · ; ξ i )]
= g i ( · ) and
E [ ∇ g i ( · ; ξ i )]
= ∇ g i ( · ) .
A special case to be considered separately is when each ξ i has a finite support and is uniformly distributed.
In this case, the problem can be represented as:
[Equation]
These problems are different from classical stochastic compositional optimization (SCO) problems
E ζ [ f ζ ( E
[Equation]
g ( w ; ξ ))] and its finite-sum variant 1 /m (cid:80)
[Equation]
[Wang
[Equation]
2017], because the inner function is coupled with the outer index in FCCO.
A striking difference in solving FCCO problems is that we need to deal with multiple functional mappings of g i ( w ) for i = 1 , . . .
, m .
A challenge emerges when it is not possible to draw data samples for all blocks i = 1 , . . .
, m at each iteration due to some restrictions (e.g., limited memory and computational budget per-iteration).
Wang and Yang [2022] studied this problem comprehensively and proposed an algorithm named as SOX.
A key to their algorithmic design is to maintain and selectively update a sequence of estimators u
= ( u 1 , . . .
, u m ) for tracking g ( w ) = ( g 1 ( w ) , . . .
, g m ( w )) by exponential moving average, i.e.,
[Equation]
[Equation]
where ξ it and B 1 t ⊆ { 1 , . . .
, m } denote a set of sampled blocks.
With u , the gradient estimator is computed by exponential moving average as well.
As a result, they establish a sample complexity of O ( mϵ − 4 ) for non-convex objectives, O ( mϵ − 3 ) for convex objectives and O ( mµ − 2 ϵ − 1 ) for µstrongly convex objectives.
However, there are several caveats of these results: (i) the sample complexities (e.g., O ( mϵ − 4 ) for a non-convex objective) are no better than probing all blocks at each iteration, for which Ghadimi et al.
[2020] have established an O ( ϵ − 4 ) iteration complexity and an O ( mϵ − 4 ) sample complexity; (ii) when m = |B 1 t | = 1 , the problem reduces to a special case of classic SCO problems; however, the complexities are worse than the state-of-the-art (SOTA) sample complexities for non-convex, convex and strongly convex objectives, which are O
( ϵ − 3 ) , O ( ϵ − 2 ) and
O ( µ − 1 ϵ − 1 ) , respectively [Zhang and Xiao, 2019, Jiang et al., 2022].
A useful technique for achieving these complexities in prior works is by using variance reduction techniques, so a straightforward approach is to change the update of u it by using a variance reduced estimator and do similarly for the gradient estimator.
In particular, one can change the update for u it according to STORM [Cutkosky and Orabona, 2019]:
[Equation]
However, this simple change does not improve the complexities over that obtained by Wang and Yang [2022].
The reason is that the standard error correction term marked above in STORM only accounts for the randomness in g i ( w t ; ξ i t ) but not in the randomness caused by sampling i ∈ B t 1 .
So, a major question remains:
How can we further improve the complexities for solving FCCO to match the SOTA results of
SCO by using variance reduction techniques via probing only O (1) blocks at each iteration?
To address this issue, we propose a novel variance reduction technique by selectively updating u it for tracking g ( w t ) , to which we refer as Multi-block-Single-probe variance-reduced (MSVR) estimator.
It employs a similar update as STORM for selected u it but with a different customized error correction term to deal with the randomness in both g i ( w t ; ξ it ) and that in B t 1 .
Based on MSVR, we develop several algorithms for FCCO problems with different ways to compute the gradients, and analyze the sample complexities across a spectrum of settings with non-convex/convex/strongly convex/PL objectives and finite/infinite support of ξ i .
We summarize our contributions and our results below:
• We develop a novel MSVR estimator for tracking a sequence of multiple blocks of functional mappings by only probing O (1) blocks via random samples at each iteration.
[Table]
[Table]
• By applying the MSVR estimator, we develop three algorithms for FCCO by using different methods for computing the gradients, and establish improved complexities for non-convex, convex, strongly convex, and PL objectives.
A comparison between our algorithms and existing methods is shown in Table 1, where we also exhibit the dependence on B 2 , which is the size of the inner batch for estimating each g
i ( w ) .
• The complexity of our first method (i.e., MSVR-v1) enjoys the same order on ϵ as SOX, but does not depend on m ; MSVR-v2 improves the dependence on ϵ , and its complexities match the SOTA results for SCO when m = 1 ; our MSVR-v3 further reduces the dependence on ϵ for the finite support of ξ , and also attains the SOTA complexities when m = 1 .
• We conduct experiments on multi-task deep AUC maximization to verify the theory and demonstrate the advantage of the proposed algorithms.
Related work
This section briefly reviews related work on variance-reduced methods and stochastic compositional optimization (SCO) problems.
Variance-reduction (VR) techniques for improving the convergence of stochastic optimization originate from Roux et al.
[2012] for solving convex finite-sum empirical risk minimization (ERM) problems.
Since then, different VR techniques have been proposed for convex finite-sum ERM, e.g.,
SVRG [Johnson and Zhang, 2013, Zhang et al., 2013] and SAGA [Defazio et al., 2014].
These works have improved the complexity for solving smooth and strongly convex problems to a logarithmic complexity.
For non-convex ERM problems, Fang et al.
[2018] invents the SPIDER estimator similar to its predecessor SARAH
[Nguyen et al., 2017], and improve the complexity of standard SGD from
O ( ϵ − 4 ) to O ( ϵ − 3 ) and O ( √ nϵ − 2 ) in stochastic and finite-sum settings, respectively, where n is the number of components in the finite-sum.
Algorithmic improvements have been made to SPIDER by using a constant step size in SpiderBoost [Wang et al., 2018] and using a constant batch size in
STORM [Cutkosky and Orabona, 2019].
Several classes of SCO have been studied.
The first class is the two-level SCO whose objective is given by E ξ [ f ξ ( E ω [ g ω ( w )])] , where ξ and ω are random variables.
While the study of two-level compositional functions dates back to the 70s, the most recent comprehensive study was initiated by
Wang et al.
They proposed a two time-scale classic algorithm named SCGD and establish its asymptotic guarantee and non-asymptotic convergence rates.
Following this work, many studies have been devoted to improving the convergence rates or algorithmic design of two-level SCO [Wang et al., 2016, Ghadimi et al., 2020, Zhang and Lan, 2021].
In particular, recent works have used variance-reduction techniques based on SPIDER/SARAH/STORM to estimate the inner values and the gradients [Liu et al., 2018, Yuan et al., 2019a, Zhang and Xiao, 2019, Chen et al., 2021, Qi et al., 2021a].
Similar efforts have been extended to the second class of SCO, i.e., multi-level SCO with an objective E ξ 1 [ f 1 ξ 1 ( E ξ 2 [ f 2 ξ 2 ( . . .
( E ξ K [ f Kξ K ( w ))] . . . )])]
[Yang et al., 2019].
Recent studies have been focused on further improving the sample complexity and reducing the dependence on the number of levels K [Balasubramanian et al., 2021, Chen et al., 2021, Zhang and Lan, 2021, Zhang and Xiao, 2021, Jiang et al., 2022].
These works also employed variance reduction techniques to design their own methods.
However, directly applying these algorithms of two-level and multi-level SCO to
FCCO requires probing all
m blocks in g ( w ) , which is prohibitive in many applications.
The third class of SCO is the Conditional Stochastic Optimization (CSO) whose objective is in the form of E ξ [ f ξ ( E ω | ξ g ω ( w ; ξ )])]
[Hu et al., 2020], where ω | ξ means that the distribution of ω might depend on ξ .
The FCCO problem can be considered as a special case of CSO.
The key difference from the first class of SCO discussed above is that the inner function g depends on the random variable ξ of the outer level.
For CSO, Hu et al.
[2020] proposed two algorithms with and without using the variance-reduction technique (SpiderBoost) named BSGD and BSpiderboost, and established complexities for non-convex, convex and strongly convex functions, which are shown in Table 1.
However, their algorithms require a large batch size for estimating the inner functions.
Recently, a novel class (the fourth class) of SCO was studied, which is referred to as the finite-sum coupled compositional optimization (FCCO)
[Wang and Yang, 2022].
The finite-sum structure makes it possible to develop more practical algorithms without relying on huge batch size per-iteration.
It was first studied by Qi et al.
[2021b] for maximizing the point-estimator of the area under the precision-recall curve.
Recently, it was comprehensively investigated by Wang and Yang [2022] and more applications of FCCO have been demonstrated in machine learning.
Nevertheless, their algorithm—SOX does not use variance reduction techniques and hence suffers from the limitations discussed in the previous section.
Proposed Algorithms and Convergence
First, we introduce the notations and assumptions used in this paper.
Then we describe the MSVR estimator in detail and develop algorithms based on the proposed estimator.
3.1 Notations and Assumptions
Let [ m ] = { 1 , . . .
, m } .
The definition of sample complexity is given below, which is widely used to measure the efficiency of stochastic algorithms.
Definition
The sample complexity is the number of samples needed to find a point satisfying E [ ∥∇ F ( w ) ∥ ] ≤ ϵ ( ϵ -stationary) or E [ F ( w ) − inf w F ( w )] ≤ ϵ ( ϵ -optimal).
Next, we make following assumptions throughout the paper, which are commonly used in the studies of SCO [Wang et al., 2016, 2017, Yuan et al., 2019a, Zhang and Xiao, 2019, 2021].
Assumption 1 (Smoothness and Lipschitz continuity)
We assume that each f i is L f -smooth and
C f -Lipchitz continuous, each g i is L g -smooth and C g -Lipschitz continuous.
Remark: This implies F ( w ) is L F -smooth, where L F = C
f L g + C 2 g L f [Zhang and Xiao, 2021].
Assumption 2 (Bounded variance)
[Equation]
where the random variable ξ it denotes a batch of samples with batch size B 2 ≥ 1 .
Assumption 3 (Average Lipchitz continuity of g i and its Jacobian)
[Equation]
Remark:
Although this assumption seems strong at the first sight, it is quite standard and widely used in the recent compositional optimization literature [Yuan et al., 2019a, Zhang and Xiao, 2019, 2021, Jiang et al., 2022].
Assumption 4 F ∗ = inf w F ( w ) ≥ −∞ and F ( w 1 ) − F ∗ ≤ ∆ F for the initial solution w 1 .
3.2 Multi-block-Single-probe Variance Reduced (MSVR) Estimator
Assume that we have a budget to probe only B 1 out of m mappings in g ( w ) .
To this end, at the t -th iteration we sample a set of blocks B t 1 ⊆
[ m ] , where |B t 1 | =
B 1 , and probe the corresponding g i ( w ) by accessing the noisy estimates g i ( w t ; ξ it ) for i ∈ B t 1 .
Then, we just update the corresponding block in our estimator u t .
Specifically, we update u it for i ∈ B t 1 in a new way and keep other blocks unchanged.
The whole estimator is shown below:
[Equation]
The first line of our estimator is inspired by STORM [Cutkosky and Orabona, 2019].
The difference is that the STORM estimator sets γ t =
(1 − β t ) , while for MSVR, γ t is set as m − B 1 B 1 (1 − β t )
+ (1 − β t ) according to our analysis.
We name equation (5) as Multi-block-Single-probe Variance Reduced (MSVR) estimator.
By multi-block, we mean the estimator can track multiple functional mappings ( g 1 , g 2 , · · · , g m ) , simultaneously; by single-probe, we indicate the number of sampled blocks Bfor probing can be as small as one.
It is notable that when B 1 = m , i.e., all blocks are probed at each iteration, γ t = 1 − β t and MSVR reduces to STORM applied to g ( w ) .
The additional factor in γ t , i.e., γ 0 t = m − B 1 B 1 (1 − β t ) is to account for the randomness in the sampled blocks and noise in those blocks that are not updated.
To briefly understand the additional factor γ 0 t , we consider bounding ∥ u t − g ( w t ) ∥ 2 = (cid:80) mi =1 ∥ u it − g
i ( w t ) ∥ 2 .
Let us focus on a fixed i ∈ [ m ] .
Then we have
E
[Equation]
Note that the first term A 1 in the R.H.S. can be bounded similarly as STORM by building recurrence with ∥ u it − 1 − g i ( w t − 1 ) ∥ 2 .
However, there exists the second term due to the randomness of B t 1 , which can be decomposed as
[Equation]
The first two terms in R.H.S. ( A 21 and A 22 ) can be easily handled.
The difficulty comes from the third term, which cannot be simply bounded by using Young’s inequality.
If doing so, it will end up with a non-diminishing error of u it .
To combat this difficulty, we use the additional factor brought by γ 0 t ( g i (cid:0) w t ; ξ it (cid:1)
− g i (cid:0) w t − 1 ; ξ it (cid:1) ) in A 1 to cancel A 23 .
This is more clear by the following decomposition of A 1 .
[Equation]
In light of the above decomposition, we can bound E [ ∥ A 11 + A 12 ∥ 2 ] ≤ E [ ∥
A 11 ∥ 2 + ∥
A 12 ∥ 2 + 2 A ⊤ 11 A 12 ] and E [ ∥ A 13 + A 14 ∥ 2 ] ≤ 2 E [ ∥
A 13 ∥ 2 ] + 2 E [ ∥ A 14 ∥ 2 ] .
The resulting term E [2 A ⊤ 11 A 12 ] has a negative sign as A 23 .
Hence, by carefully choosing γ 0 t , we can cancel both terms.
The remaining terms can be organized similarly as in the analysis for STORM.
We give a technical lemma for building the recurrence of MSVR’s error below.
All the proofs are deferred to the supplementary material due to space limitations.
Lemma
By setting γ t = m − B 1 B 1 (1 − β t ) + (1 − β t ) , for β t ≤ 1 2 , we have:
[Equation]
1: Input do 3:
Compute estimator u to equation
Compute
STORM
8: end
9: Choose
10: Return
Remark:
The above recursion is similar to that of STORM for tracking a sequence of a single-block functional mapping.
Since the last term ∥ w t − w t − 1 ∥ 2 can be offset in the future analysis, intuitively the estimation error ∥
u t − g ( w t ) ∥ 2 would reduce after each iteration.
Single Point Version.
A limitation of the MSVR estimator is that it needs to probe selected blocks at two different points, i.e., g i ( w t ; ξ it ) and g i ( w t − 1 ; ξ it ) .
With a more careful analysis, we can probe a selected block at a single point similar to that used by Balasubramanian et al.
[2021] and Chen et al.
Specifically, we replace g i (cid:0) w t ; ξ it (cid:1)
− g i (cid:0) w t − 1 ; ξ it (cid:1) with ∇ g i (cid:0) w t ; ξ it (cid:1) ⊤ ( w t − w t − 1 ) .
As a result, we propose a single-point version of MSVR (named as MSVR-SP) estimator below:
[Equation]
The MSVR-SP estimator enjoys the similar recurrence for the estimation error.
Lemma 2 Set γ t = m
[Equation]
B 1 B 1 (1 − β t )
+ (1 − β t ) .
If ∥ w t +1 − w t ∥ 2 ≤ η 2 t L 2 F and η t ≤ √ β t , we have:
[Equation]
Remark:
If there is a constraint on the range of g i , we can add a projection to the update of u it such that it always resides in the range, which will not affect the analysis of Lemma 1 and Lemma 2.
3.3 Leveraging the MSVR Estimator for solving the FCCO Problem
Now, we are ready to present our proposed algorithms for solving problem (1).
The first two algorithms (named MSVR-v1 and MSVR-v2) are presented in Algorithm 1.
These two methods differ in how to estimate the gradient.
Let us first consider MSVR-v1.
At each time step t , we first use the proposed MSVR or MSVR-SP estimator u t to estimate the inner function value.
Then, following the previous literature [Wang et al., 2021, Wang and Yang, 2022], we use the moving average estimator z t to estimate the gradient as:
[Equation]
where Π L F denotes the projection onto the ball with radius L F .
This projection is optional for using
MSVR, but is required for using MSVR-SP to ensure ∥ w t +1 − w t ∥ 2 ≤ η 2 t L 2 F as used in Lemma 2.
Since the true gradient ∇ F is also in this ball, i.e., ∥∇ F ∥ ≤ L F , the projection will not affect the future analysis.
Also note that when computing the estimator z t , we use ∇ f i ( u it − 1 ) instead of ∇ f i ( u it ) to avoid the dependence on the random variable ξ it , which may lead to dependent issues otherwise.
Finally, we use the estimated gradient z t to update the parameter w t +1 .
Now, we provide the theoretical guarantee for the MSVR-v1 method.
Theorem 1 Our MSVR-v1 algorithm with α t +
=
O ( η t ) , β t +
=
O ( m 2 η 2 t B 21 ) , a = O ( mB 2 B 1 ) and η
[Equation]
= min (cid:26)(cid:16)
[Equation]
, can find an ϵ -stationary point in O
[Equation]
Remark:
This complexity is strictly better than previous SOTA method SOX, which enjoys an iteration complexity of O (cid:16) max (cid:110)
[Equation]
The sample complexity can be obtained by multiplying the iteration complexity with B 1 B 2 .
We can see that larger B 1 or B 2 yields a smaller iteration complexity, which means that from the computational perspective, if samples can be processed in parallel (e.g., in GPU), there is a benefit of using large B 1 and/or B 2 .
However, from the sample complexity perspective, using B 1 = B 2 = 1 is the best.
The same discussion holds for other theorems below.
However, the complexity of MSVR-v1 is still on the order of O ( ϵ − 4 ) .
Due to the biased nature of the estimated gradient, using the moving average update is not enough for achieving the SOTA complexity of O ( ϵ − 3 ) .
So, we use the technique of STORM [Cutkosky and Orabona, 2019] to update z t as follows:
[Equation]
where the projection operation is needed if using the MSVR estimator.
Now, we prove this new method (i.e., MSVR-v2) can obtain the optimal complexity of O ( ϵ − 3 ) .
Theorem 2 Our MSVR-v2 algorithm with α t +
= O ( mη 2 t B 1 ) ,
β t +1 =
O (cid:16)
m 2 η 2 t B 21 (cid:17) , a = O ( mB 2 B 1 ) and η t = O (cid:16) ( B 1 √ B 2 m )
[Equation]
/ 3 ( a + t ) −
[Equation]
3 (cid:17) , can find an ϵ -stationary point in O (cid:16) mϵ − 3 B 1 √ B 2 (cid:17) iterations.
Remark: When m = 1 and f is the identity function, problem (1) reduces to the standard stochastic non-convex optimization, whose lower bound is Ω (cid:0) ϵ − 3 (cid:1)
[Arjevani et al., 2019], indicating our
MSVR-v2 is optimal.
Next, we show that the complexity can be further improved when the objective function is convex or strongly convex.
We note that Polyak-Łojasiewicz (PL) [Karimi et al., 2016] objectives are more general than strongly convex functions, since µ -strong convexity implies the µ -PL condition.
So, we will consider the PL condition and introduce its definition below.
Definition 2 F ( w ) satisfies the µ -PL condition if there exists µ > 0 such that:
[Equation]
Then, we derive improved rates for convex or PL objectives by using the stage-wise design given in
Algorithm 3 in the supplement.
Theorem
If the objective function satisfies the convexity or µ -PL condition, MSVR-v1 derives a sample complexity of O (max( B 1 , B 2 )
ϵ − 3 ) or O (max( B 1 , B 2 ) µ − 2 ϵ − 1 ) , separately.
For MSVR-v2, the complexity can be further improved to O (cid:0) m √ B 2 ϵ − 2 (cid:1) or O (cid:0) m √ B 2 µ − 1 ϵ − 1 (cid:1) .
Remark: The complexities for MSVR-v2 are optimal, since they match the Ω (cid:0) ϵ − 2 (cid:1) and Ω (cid:0) µ − 1 ϵ − 1 (cid:1)
lower bound for stochastic convex and strongly convex optimization [Agarwal et al., 2012].
Remark:
The algorithms proposed in this paper can also use adaptive (Adam-style) learning rates and obtain the same complexity using the techniques proposed by Guo et al.
The details are provided in the supplementary.
Sample a
Compute function value estimator u t according equation
Compute gradient z t
11: end
12: Choose uniformly
13: Return w τ
An Improved Rate for the Finite-sum Case
In this section, we consider the case that inner function
g
i is in the form of the finite-sum, i.e., g i ( w ) = 1 n (cid:80) nj =
g i ( w ; ξ ij ) , so that we can compute the exact value of g i ( w ) in some iterations.
We first modify our MSVR estimator to utilize the finite-sum structure.
Inspired by SVRG [Johnson and
Zhang, 2013, Zhang et al., 2013], we compute a full version of the inner function value for every I iterations at w τ , i.e., g i ( w τ ) =
1 n (cid:80) nj =
g
i ( w τ ; ξ ij ) for i = 1 , · · · , m , where τ mod I = 0 .
Then, in each step, we use
[Equation]
to replace g i ( w t ; ξ it ) in the origin estimator.
In this way, our MSVR estimator is changed to:
[Equation]
For this estimator, we have the following guarantee.
Lemma 3 If β ≤ 12 and βI ≤ mB 1 , by setting γ = m − B 1 B 1 (1 − β ) + (1 − β ) , we have:
[Equation]
Remark:
Compared with Lemma 1, we remove the 2 B 1 β 2 σ 2 Bterm, which is the key to reduce the complexity since we can now use a larger parameter β .
To attain the optimal complexity, we modify the gradient estimator z t in a similar way:
[Equation]
where h t involves both the full gradient and the stochastic gradient (we also need to save each ∇ f i ( u τ − 1 ) and calculate the full version of 1 m (cid:80) mi =1 ∇ f i ( u iτ − 1 ) ∇ g
i ( w τ ) at those steps τ )
[Equation]
The whole method is summarized in Algorithm 2 (named as MSVR-v3).
Next, we show that
MSVR-v3 is equipped with an optimal complexity of O ( √ nϵ − 2 ) .
Theorem 4 Our MSVR-v3 with I = mnB 1 B 2 , α = O (cid:0) B 1 B 2 mn (cid:1) , β
= O (cid:0) B 2 n (cid:1) and η = O (cid:16) B 1 √ B 2 m √ n (cid:17) , can obtain an ϵ -stationary point in T
= O (cid:16) m √ nϵ − 2 B 1 √ B 2 (cid:17) iterations.
[Figure]
Remark: When m = 1 and f is the identity function, problem (2) reduces to the stochastic finite-sum optimization, whose optimal complexity is O (cid:0) √ nϵ − 2 (cid:1)
[Fang et al., 2018, Li et al., 2021], indicating our complexity is optimal in terms of ϵ and n .
Similarly, a better complexity can be obtained under the convexity or PL condition.
Theorem
If the objective function satisfies the convexity or µ -PL condition, the sample complexity can be improved to O (cid:16) m √ nϵ − 1 B 1 √ B 2 log 1 ϵ (cid:17) or O (cid:16) m √ nµ − 1 B 1 √ B 2 log
1 ϵ (cid:17) , respectively.
Remark: It is notable that we achieve a linear convergence rate O (cid:0) log 1 ϵ (cid:1) under the PL condition, matching the current result in the single-level finite-sum problem [Li et al., 2021]
Experiments
In this section, we conduct experiments on the multi-task deep AUC maximization to evaluate the proposed methods and we will consider more applications in the long version of the paper.
For binary classification (label y = 1 or y = − 1 ), AUC maximization can be formulated as minimizing the following composite loss [Zhu et al., 2022]:
[Equation]
where a ( w ) =
E [ h w ( x ) | y = 1] , b ( w ) =
E [ h w ( x ) | y = − 1] and ℓ ( · ) is a surrogate function.
The above objective recovers the pairwise square loss and the min-max margin loss proposed by Yuan et al.
[2020] for deep AUC maximization by setting ℓ ( · ) as the square function or squared hinge function, respectively.
When applied to multi-task classification (e.g., multiple classes), we can optimize the averaged AUC losses over all tasks, i.e., AUC = 1 m (cid:80) mi =1 AUC ( i ) .
The nested structure only comes from the term ℓ ( a ( w ) − b ( w )) , and we can rewrite it as the form of FCCO problem, where
[Equation]
where D i + / − denots the positive/negative datasets of the i -th task.
Configurations.
In the experiment, we follow the setup in Zhu et al.
[2022] and set the surrogate function ℓ as squared hinge ℓ ( x ) =
12 (max { c + x, 0 } ) 2 .
We use ResNet18 as backbone network, and train on six datasets:
STL10 [Coates et al., 2011], CIFAR10 [Krizhevsky, 2009], CIFAR100 [Krizhevsky, 2009], MNIST [LeCun et al., 1998], Fashion-MNIST [Xiao et al., 2017], and SVHN [Netzer et al., 2011].
We compare our methods with previous SOTA algorithm SOX
[Wang and Yang, 2022].
For our methods, parameters α and β are searched from { 0 . 1 , 0 . 5 , 0 . 9 , 1 . 0 } .
For SOX algorithm, its parameters β and γ are searched from the same set.
B 1 is set as 50 for CIFAR100 and 5 for other datasets.
Inner batch size B 2 is chosen as 128 for all methods.
We tune the learning rate from the set { 1 e − 4 , 1 e − 3 , 2 e − 3 , 5 e − 3 , 1 e − 2 } and pick the best one for each method.
The experiments are conducted on single NVIDIA Tesla M40 GPU.
Results.
Figure 1 shows the loss against the number of samples drawn by different methods, and all curves are averaged over 5 runs.
We observe that MSVR-V1 is better than SOX on the CIFAR dataset, and close to it on other datasets.
MSVR-v2 converges faster than SOX and MSVR-v1, and the loss of MSVR-v3 decreases most rapidly, demonstrating a low sample complexity.
Conclusion and Future Work
In this paper, we develop a novel MSVR estimator for tracking multiple functional mappings by probing only O (1) blocks.
Equipped with this estimator, we design three algorithms for FCCO problems and obtain improved complexities across a spectrum of settings.
Experimental results on multi-task deep AUC maximization also verify the effectiveness of our methods.
In future work, we will investigate other applications that can be solved by using the proposed estimator.
Acknowledgments and Disclosure of Funding
W. Jiang, Y. Wang and L. Zhang were partially supported by NSFC (62122037, 61921006).
G. Li and T. Yang were partially supported by Amazon research award.
The authors would like to thank the anonymous reviewers for their helpful comment.
A Analysis A.1 Proof of Lemma
[Equation]
For the first term
[Equation]
we can decompose it as:
[Equation]
Also, the second term can be written as:
[Equation] [Equation]
=
[Equation] [Equation]
ξ
[Equation]
ξ it
[Equation] [Equation]
ξ
[Equation] [Equation]
ξ
[Equation] [Equation]
(cid:105)
[Equation]
= B
[Equation]
The second equation is because of E (cid:2) g i ( w t +1 ; ξ it +1 )
− g i ( w t +1 ) (cid:3) = 0 .
The last equation is due to
E (cid:2) g
i
[Equation]
w t +1 ; ξ it +1 )
[Equation]
g i
[Equation]
it
[Equation] [Equation]
= g
[Equation] [Equation]
i ( w t ) and u it is independent of ξ it +1 .
We want to ensure 1 ⃝ + 2 ⃝ = 0 , which requires that 2(1 − B 1 m ) + 2 B 1 m (1 − β t +1 )(1 − β t +1 − γ t +1 )
= 0 .
Solve γ t +
and we have γ t +
= m − B 1 B 1 (1 − β t +1 )
+ (1 − β t +1 ) .
According to equation (12) and equation (13), the equation (11) can now be written as:
[Equation]
The second equation is due to
E
[Equation] [Equation]
it
[Equation] [Equation]
i
[Equation]
it
[Equation] [Equation]
The first inequality is because of γ t +1 ≤ 2 mB 1 (since β t +1 ≤ 12 ) and 1 − B 1 m + B 1 m (1 − β t +1 ) 2 ≤ 2 B 1 m (1 − β t +1 ) γ t +1 .
The last inequality is due to the fact that
[Equation]
Finally
[Equation]
we have:
[Equation]
A.2 Proof of Lemma
Proof We will start with single block and them sum over multiple blocks.
To start, we have
[Equation]
Next, we will proceed to decompose A
[Equation]
By taking expectation over A
[Equation]
we have
[Equation]
w
[Equation] [Equation]
[Equation] [Equation]
(cid:21)
Since γ t +1 + β t +1 − 1 = m − B 1 B 1 (1 − β t +1 ) , the terms involving A 0 , A 1 will cancel.
As a result, we have
[Equation]
Since ∥ ∆ t ∥ ≤ min( L g 2 ∥
w t +1 − w t ∥ 2 , 2 C g ∥
w t +1 − w t ∥ ) and ∥ g i ( w t +1 )
− g i ( w t ) ∥ ≤ C g
∥ w t +1 − w t ∥ 2 , we have
[Equation]
−
[Equation]
∥
[Equation]
(cid:21)
Note that we have ∥ w t +1 − w t ∥ 2 ≤ η 2 t +1 L 2 F , γ t +1 ≤ 2 mB 1 and η t +1 ≤ (cid:112) β t +1 .
Therefore
E
[Equation]
u
[Equation]
+1 − g
i
[Equation]
w
[Equation] [Equation]
1 −
[Equation]
Finally
[Equation]
we have:
[Equation]
A.3 Proof of Lemma
Proof Note that we have:
[Equation]
Since τ is the closest small index such that τ mod I =
[Equation]
we have:
[Equation]
We can then apply the same analysis as in Section A.1, until equation (14):
[Equation]
The last inequality is due to βI ≤ mB 1 .
Finally, we have:
[Equation]
A.4 Proof of Theorem
We denote constant C = max (cid:110) 1 , C 2 g , L 2 F , σ 2 , L 2 f C 2 g , L 2 g C 2 f , L 2 f C 4 g , L 2 f C 2 g σ 2 , C 2 f ( σ 2 + C 2 g ) (cid:111) .
Lemma 4 (Lemma 2 in Li et al.
Suppose function F is L F -smooth and consider the update w t +1 :
= w t − η t z t .
With η t L ≤ 12 , we have:
[Equation]
Lemma
Denote ∥
u t − g ( w t ) ∥
= (cid:80) mi =1 ∥
u it − g i ( w t ) ∥ 2 and ∥
u t − u t − 1 ∥ 2 = (cid:80) mi =1 ∥
u it − u
it − 1 ∥ 2 .
E (cid:2) ∥
[Equation]
Proof According to Lemma 1 in Wang and Yang [2022]
[Equation]
if α ≤
[Equation]
we have:
[Equation]
By setting α ≤
[Equation]
we have the above lemma.
Lemma 6 If β t +1 ≤
[Equation]
we have:
[Equation]
Proof Note that with β t +1 ≤ 12 , we have γ t +1 ≤ 2 mB
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The third inequality is due to E (cid:2)
g i ( w t +1 ; ξ it +1 )
− g i ( w t +1 ) (cid:3) = 0 .
The rest proof of Theorem 1 Let Γ t =
F ( w t ) +
[Equation]
By setting η t = 2 α t +1 c 0 , C 0 = 72 C , η t ≤ B 1 4 m we have:
[Equation]
By setting β t +1 =
m 2 C 2 η 2 t B(and note that c 0 = 72 C , α t +1 =
Cη t ), we have:
[Equation]
This means that, by setting η t = min { (cid:112)
[Equation]
B
[Equation]
B
[Equation]
η T
[Equation]
Similar to the proof of Theorem 1 in STORM [Cutkosky and Orabona, 2019],
denote M = ∆ F + 2 C c
η 0 + 16 3 C 4 ln (1 + T ) .
Using Cauchy-Schwarz inequality, we have:
[Equation]
, which indicate that
[Equation]
Finally, using Cauchy-Schwarz we have (cid:80) T t =1 ∥∇ F ( w t ) ∥ /T ≤
[Equation]
T t =1 ∥∇ F ( w t ) ∥ 2 / √ T
so that:
E
[Equation]
max
[Equation]
≤ max
[Equation]
where the last inequality is due to ( a + b ) 1 / 3 ≤ a 1 / 3 + b 1 / 3 .
So, we can achieve the stationary point with T
= O (cid:16) max (cid:110)
[Equation]
Lemma
[Equation]
Proof
[Equation]
≤ E
[Equation]
[Table]
2 (cid:35)
Lemma
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
z t − 1 m m (cid:88) i =1 ∇ g
i ( w t ) ∇ f i ( u it − 1 ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   ≤ E (cid:34) (1 − α t ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
z t − 1 − 1 m m (cid:88) i =1 ∇ g
i ( w t − 1 ) ∇ f i ( u it − 2 ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
+
[Equation]
α
[Equation]
t σ
B+
[Equation]
C
[Equation]
g L 2 f m ∥
u t − 1 − u t − 2 ∥ 2 + 4 C 2 f L 2 g ∥ w t − w t − 1 ∥(cid:35)
Proof
E (cid:34) ∥
z t − m m (cid:88) i =1 ∇ g i ( w t ) ∇ f i ( u it − 1 ) ∥(cid:35) =
E (cid:34)(cid:13)(cid:13)(cid:13)(cid:13) (1 − α t ) (cid:32) z t − 1 − 1 m m (cid:88) i
=1 ∇ g
i ( w t − 1 ) ∇ f i ( u it − 2 ) (cid:33) + α t   1 B 1 (cid:88) i ∈B t 1 ∇ g
i ( w t ; ξ it ) ∇ f i ( u it − 1 ) − 1 m m (cid:88) i
=1 ∇ g
i ( w t ) ∇ f i ( u it − 1 ) 
 +(1 − α t )  
[Equation]
We assume that E
[Equation]
Due to the fact that the expectation over the last two terms equals zero
[Equation]
we have:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Lemma 9 Suppose that β ≤ 132 C and B 1 β t +1 ≤ mα t +1 .
Then, we have:
[Equation]
u it ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   ≤ (1 −
B 1 β t + m )m E (cid:104)
∥
u t − g ( w t ) ∥ 2 (cid:105) + 8 mC 2 g
B∥ w t
+1 − w t ∥ 2 + 2 B 1 β 2 t +1 σ
B 2 m + (1 − α t +1 )
E   (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)
z t − 1 m m (cid:88) i =1 ∇ g
i ( w t ) ∇ f i ( u it − 1 ) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) 2   + 2 α 2 t +1 σ
B+ 4 C 2 g L 2 f m
E (cid:2)
∥
u t − u t − 1 ∥ 2 (cid:3) + 4 C 2 f
L 2 g E (cid:2)
∥ w t +1 − w t ∥ 2 (cid:3) ≤ (1 −
B 1 β t + m ) E   1 m ∥
u t − g ( w t ) ∥ 2 + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
z t − 1 m m (cid:88) i =1 ∇ g
i ( w t ) ∇ f i ( u it − 1 ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   + 12 mC
B∥ w t
+1 − w t ∥ 2 + 2 B 1 β 2 t +1 σ
B 2 m + 2 α 2 t
+1 σ
B+ 4 C 2 g L 2 f m
E (cid:2)
∥
u t − u t − 1 ∥ 2 (cid:3) ≤ (1 −
B 1 β t +
[Equation]
The rest proof of Theorem 2 Set η t ≤ B 1 mc 0 .
Denote Γ t = F ( w t )
+
B 1 c 0 η t m ∆ t , where ∆ t =
[Equation]
m ∥ u t − g
[Equation]
w
[Equation]
+
[Equation]
We have:
[Equation]
By setting 264 C =
[Equation] [Equation]
t +1 =
B
[Equation]
t
[Equation] [Equation]
and B 2 ≤ m , we have:
[Equation]
This means that, by setting η
[Equation]
Denote M = ∆ F + 18 η 0 + c 40
ln (1 + T ) .
Using Cauchy-Schwarz inequality
[Equation]
we have:
[Equation]
, which indicate that
[Equation]
Finally, using Cauchy-Schwarz we have (cid:80) Tt =
1 ∥∇ F ( w t ) ∥ /T ≤ (cid:113)(cid:80)
Tt =1 ∥∇ F ( w t ) ∥ 2 / √ T
so that:
[Equation]
where the last inequality is due to ( a + b ) 1 / 3 ≤ a 1 / 3 + b 1 / 3 .
So, we can achieve the stationary point with T
= O (cid:0) m/B 1 √ B 2 ϵ 3 (cid:1) .
Input initial points w 0 , u 0 , z 0 ) for
1 to S do w
A.6 Proof of Theorem
We would show that the complexity can be further improved if the objective function satisfies the
Polyak-Łojasiewicz (PL) condition or convexity.
To achieve this, we utilize the previous analysis and use a stage-wise version method [Yuan et al., 2019b].
In the new algorithm, we decrease α s and β s after each stage and increase the number of iterations T s .
At the end of each stage, we save the output and use it to restart the next stage.
With these modifications, we can obtain a better convergence guarantee under the PL condition or convexity.
The new method is summarized in Algorithm 3, named Stage-wise MSVR.
Next, we will show the proof for optimal MSVR-v2 with Stage-wise version, and the proof for MSVR-v1 is nearly the same as the MSVR-v2.
Note that in below the numerical subscripts denote the stage index { 1 , . . .
, S } .
Denote ∆ s = (cid:13)(cid:13)
z s − 1 m (cid:80) mi =1 ∇ g
i ( w s ) ∇ f i ( u is − 1 )
(cid:13)(cid:13)
+ 1 m ∥ u s − g ( w s )
∥ 2 .
Let’s consider the first stage, ∆ 1 ≤ 2 C = µϵ 1 and F ( w 1 ) − F ∗ ≤ ϵ 1 , where ϵ 1 = max { 2 Cµ , ∆ F } .
Starting form the second stage, we would prove by induction.
Suppose at stage s − 1 , we have ∆ s − 1 ≤ µϵ s − 1 and F ( w s − 1 ) − F ∗ ≤
ϵ s − 1 .
Then at s stage, by setting 264 C = c 0 , η 2 s = 32 B 21 β s m 2 c 20 , α s =
B 1 β s m and B 2 ≤ m , we have:
[Equation]
This means that by setting T s = max (cid:110)
[Equation]
, we have:
[Equation]
Due to the PL condition
[Equation]
we have:
[Equation]
On the other hand
[Equation]
by setting β s = B 2 µϵ s 80 C
and α s =
B 1 β s m
[Equation]
we have:
[Equation]
So, we proved that F ( w s ) − F ∗ ≤ ϵ s .
That is to say, F ( w s ) − F ∗ ≤ ϵ when S
= log 2 (cid:0) 2 ϵ 1 ϵ (cid:1) , and the iteration complexity is computed as:
[Equation]
When F ( w ) is convex, we define ˆ F ( w ) =
F ( w ) + µ 2 ∥ w ∥ 2 .
We know that ˆ F ( w ) is µ -strongly convex, which implies µ -PL condition.
We have proved: for any δ > 0 , there exist T
= O (cid:16) mµδ (cid:17) such that ˆ F ( w T ) − ˆ F ∗ ≤ δ .
It indicates that F ( w T ) − F ∗ ≤ δ + µ 2 ∥ w ∗ ∥ 2 − µ 2 ∥ w T ∥ 2 ≤ δ + µ 2 D .
For any ϵ > 0 , if we choose µ = ϵD and δ = ϵ 2 , we get F ( w T ) − F ∗ ≤
ϵ , for some T
= O (cid:0)
mϵ 2 (cid:1) .
A.7 Proof of Theorem
Lemma 10 If β ≤ 12 and βI ≤ mB 1 , we have:
[Equation]
Proof Following the analysis of Lemma 6, we have:
[Equation]
So, we have:
[Equation]
So, with β 2 I 2 ≤ m 2 /B 21 , we have:
[Equation]
We can also replace Lemma 8 with following lemma
[Equation]
Lemma 11 With αI ≤ 1 , we have:
[Equation]
Proof First, since h t is an unbiased estimation of 1 m (cid:80) mi =1 ∇ g i ( w t ) ∇ f i ( u it − 1 ) , we have:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Next, we have:
[Equation]
E
[Equation] [Equation]
Summing up, we have:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
≤
[Equation]
The last inequality is due to αI ≤ 1 .
The rest proof of Theorem 4 According to Lemma 7, we have:
[Equation]
We use Lemma 11 to replace (cid:80) Tt
=1 (cid:13)(cid:13)
z t − 1 m (cid:80) mi =1 ∇ g i ( w t ) ∇ f i ( u it − 1 ) (cid:13)(cid:13) 2 :
E (cid:34) T (cid:88) t =1 ∥
[Equation]
[Table]
m
E
[Equation]
=
[Equation]
(cid:35)
We use Lemma 3 to replace E (cid:104)(cid:80)
Tt =1 ∥
u t − g ( w t ) ∥ 2 (cid:105) :
E (cid:34) T (cid:88) t =1 ∥
[Equation]
≤
[Equation]
m
[Equation]
i =
[Equation]
∇ g
[Equation]
w
[Equation]
f i ( u i 0 ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + 428 mC αB
T (cid:88) t =1 ∥ w t +1 − w t ∥+ 146 C 2 g L 2 f E (cid:104) ∥
u 1 − g ( w 1 ) ∥ 2 (cid:105)
B 1 β + 1460 m 2 C 4 g L 2 f
B 21 β
T (cid:88) t =1 ∥ w t +1 − w t ∥≤
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
z
[Equation]
m m (cid:88) i =1 ∇
[Equation]
f i
[Equation]
i 0 )
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
+ 146 C 2 g L 2 f E (cid:104)
∥
u 1 − g ( w 1 ) ∥
[Equation]
(cid:105)
B 1 β + 1888 m 2 C
B 21 β
[Equation]
t =1 ∥
w t +1 − w t ∥≤ ∆αT 0 + ∆βT 0 + 1888 m 2 C
B 21 β
T (cid:88) t =1 ∥ w t +
[Equation]
Set 1888 m
[Equation]
2 B
[Equation]
≤ 12 .
We have:
[Equation]
According to Lemma 4, we have:
[Equation]
Finally,
[Equation]
Note that the sample complexity is (cid:0) B 1 B 2 T + mnTI (cid:1) .
To ensure the first term and the second term at the same order, we set I = (cid:16) mnB 1 B 2 (cid:17) .
Also, since we assume that αI ≤ 1 and βI ≤ mB 1 , we directly set α = B 1 B 2 mn and β = B 2 n .
This setting also satisfies the requirement B 1 β ≤ mα .
We also require 1888 m 2 Cη 2 B 21 β ≤ 12 .
So, we set η = O ( B√ B 2 m √ n ) .
With T =
O (cid:16) m √ n B 1 √ B 2 ϵ 2 (cid:17) and T
= O (cid:16) √ n √ B 2 (cid:17) , We have: 1 T E (cid:104)(cid:80)
[Equation]
=
[Equation]
F (
[Equation]
(cid:105) ≤
[Equation]
A.8 Proof of Theorem
The analysis is very similar form Theorem 3.
We still use Algorithm 3 but employ MSVR-v3 instead.
Also, we do not need to decrease α , β , η and increase T during each stage.
Let’s consider the first stage,
[Equation]
and
F
[Equation]
≤ ∆
[Equation]
≤ ϵ 1 , where we set ϵ 1 = max { ∆ F , 146 Cµ } .
Note that in below the numerical subscripts denote the stage index { 1 , . .
. , S } .
Set α = B 1 B 2 mn , β
= B 2 n , η = O ( B 1 √ B 2 m √ n ) and
T =
O (cid:16) max (cid:110)
[Equation]
, m
[Equation]
2 (cid:111)(cid:17) .
Starting form the second stage, we would prove by induction.
Suppose at the stage s − 1 , we have F ( w s − 1 ) − F ∗ ≤ ϵ s − 1 , 4 (cid:13)(cid:13)
z s − 1 − 1 m (cid:80) mi =1 ∇ g
i ( w 1 ) ∇ f i ( u is − 2 ) (cid:13)(cid:13) 2 ≤ µϵ s − 1 , and 146 Cg L 2 f m ∥
u s − 1 − g ( w s − 1 ) ∥ 2 ≤ µϵ s − 1 .
Then at s stage, we have:
[Equation]
On the other hand, following the very similar analysis in Theorem 3, we have:
[Equation]
We proved that F ( w s ) − F ∗ ≤ ϵ s .
That is to say, F ( w S ) − F ∗ ≤ ϵ when S
= log 2 (cid:0) 2 ϵ 1 ϵ (cid:1) = log 2 (cid:0) Lϵ (cid:1) , and the iteration complexity until this stage is computed as:
[Equation]
When F ( w ) is convex, we define ˆ F ( w ) =
F ( w )+ µ 2 ∥ w ∥ 2 .
We know that ˆ F ( w ) is µ -strongly convex, which implies µ -PL condition.
We have proved: for any δ > 0 , there exist T
= O (cid:16) m √ n µB 1 √ B 2 · log 1 ϵ (cid:17) such that
ˆ F ( w T ) − ˆ F ∗ ≤ δ .
It indicates that F ( w T ) − F ∗ ≤ δ + µ 2 ∥ w ∗ ∥ 2 − µ 2 ∥ w T ∥ 2 ≤ δ + µ 2 D .
For any ϵ > 0 , if we choose µ = ϵD and δ
= ϵ 2 , we get F ( w T ) − F ∗ ≤
ϵ , for some T =
O (cid:16) m √ n ϵB 1 √ B 2 · log 1 ϵ (cid:17) .
B MSVR with Adaptive Learning Rates
Now we show that the proposed MSVR method can be extended to adaptive learning rates and remains the same sample complexity.
To use adaptive learning rates, we can revise the weight update step w t +1 =
w t − η t z t in origin MSVR method as follows:
[Equation] [Equation]
where δ > 0 is a parameter to avoid dividing zero, Π L f denotes the projection onto the ball with radius L f and h t = h ′ t (Adam-style) or h t = max ( h t − 1 , h ′ t ) (AMSGrad-style).
Inspired by the recent study of Adam-style methods [Guo et al., 2021], we can give the sample complexity of the
Adaptive MSVR using similar analysis.
We show the proof of adaptive MSVR-v2 for example:
Theorem 6 If we choose parameters α t +
= O ( mη 2 t B 1 ) ,
β t +1 =
O (cid:16)
m 2 η 2 t B 21 (cid:17) , a = O ( mB 2 B 1 ) and η t = O (cid:16) ( B 1 √ B 2 m ) 2 / 3 ( a + t ) − 1 / 3 (cid:17) , Adaptive MSVR-v2 with learning rate defined in (17), can obtain a stationary point in O (cid:16) mϵ − 3 B 1 √ B 2 (cid:17) iterations.
Remark:
The sample complexity is still at the order of O (cid:0) ϵ − 3 (cid:1) .
For MSVR-v1 and MSVR-v3, or under the convexity or PL condition, adaptive method can still get the same complexity as the origin rate using a very similar analysis.
Proof Note that since the norm of estimated gradient ∥ z t ∥ is bounded, the value of the learning rate scaling factor c = 1 / (cid:0) √ h t + δ (cid:1) is also upper bounded and lower bounded, which can be presented as c l ≤ ∥ c ∥ ∞ ≤ c u .
(Note that projection onto a ball of radius L F does not change the analysis, since ∇ F is also in this ball.)
With this property, We have:
Lemma 12 (Lemma 3 in [Guo et al., 2021])
For w t +1 = w t − ˜ η t z t , with η t c l ≤ ˜ η t ≤ η t c u and η t L F ≤ c l / 2 c 2 u , we have following guarantee:
[Equation]
Then very similar to the proof to Theorem 2.
Denote Γ t = F ( w t )
+
B 1 c
[Equation]
1 m ∆ t , where ∆ t = + 1 m ∥ u t − g ( w t ) ∥ 2 + (cid:13)(cid:13)
z t − 1 m (cid:80) mi =1 ∇ g i ( w t ) ∇ f i ( u it − 1 ) (cid:13)(cid:13) 2 .
We have:
[Equation]
By setting 256 Cc u /c l = c 0 , η 2 t = 32 B
[Equation]
and B 2 ≤ m , we have:
[Equation]
This means that, by setting η
[Equation]
[Figure] [Figure]
Denote M = ∆ F c l +
[Figure]
η 0 c l + c 40 c 2 l 16 5 ln (1 + T ) .
Using Cauchy-Schwarz inequality, we have:
[Equation]
Then following the same analysis, we will finally have :
[Equation]
where the last inequality is due to ( a + b ) 1 / 3 ≤ a 1 / 3 + b 1 / 3 .
So, we can achieve the stationary point with T
= O (cid:0) m/B 1 √ B 2 ϵ 3 (cid:1) .
C More Experimental Results
In this section, we provide more experimental results and ablation studies.
We will consider more applications in the long version of the paper.
C.1 Ablation Study on Algorithm Design
In this subsection, we conduct the ablation study for our algorithm design.
Specially, we verify the effects of our customized error correction term.
To compare with traditional variance reduced estimator, we can design an estimator using STORM [Cutkosky and Orabona, 2019] as follows:
[Equation]
[Figure] [Figure]
T r a i n i ng l o ss
B 2 = 32 B 2 =
B 2 =(c) MSVR-v
To show the effects of our customized error correction term, we replace the MSVR estimator in our
MSVR-v1 and MSVR-v2 algorithm, and use equation (18) instead.
We name these two methods
Variant-v1 and Variant-v2.
For the finite-sum case, we modify the estimator similarly:
[Equation]
where (cid:98) g i ( w t ; ξ it ) =
g i ( w t ; ξ it ) − g i ( w τ ; ξ it )
+ g i ( w τ ) .
So, for MSVR-v3, we replace the MSVR estimator with equation (19) and keep other parts unchanged.
This new method is named as Variantv3.
Results.
We compare different methods on the CIFAR100 dataset and plot the results in Figure 2.
As can be seen, all methods perform worse than the origin algorithms, indicating the effectiveness of our customized error correction term in the proposed algorithm.
C.2 Results with Different Networks
In this subsection, we conduct experiments on SVHN data set with different networks, ResNet18,
ResNet34 and DenseNet121, respectively.
As can be seen in Figure 3, with all three networks,
MSVR-V1 performs closely to SOX, MSVR-v2 converges faster than SOX and MSVR-v1, and the loss of MSVR-v3 decreases most rapidly, indicating the effectiveness of our methods with different networks.
C.3 Results with Different Batch size
In this subsection, we explore the effect of different batch sizes.
First, we fix the inner batch size
B 2 = 128 and vary B 1 in the range { 2 , 5 , 9 } .
Then, we fix the outer batch size B 1 = 5 and vary Bin the range { 32 , 64 , 128 } .
We conduct the experiments on the Fashion-MNIST data set and show the results in Figure 4 and 5.
As can be seen, in terms of iteration complexities, the larger batch size ( Bor B 2 ), the faster the convergence, which is consistent with our theory.