Multi-block-Single-probe Variance Reduced Estimator for Coupled Compositional Optimization
Abstract
Variance reduction techniques such as SPIDER/SARAH/STORM have been extensively studied to improve the convergence rates of stochastic non-convex optimization, which usually maintain and update a sequence of estimators for a single function across iterations.
What if we need to track multiple functional mappings across iterations but only with access to stochastic samples of O (1) functional mappings at each iteration?
There is an important application in solving an emerging family of coupled compositional optimization problems in the form of (cid:80) mi =
f
i
g i ( w )) , where g i is accessible through a stochastic oracle.
The key issue is to track and estimate a sequence of g ( w )
=
( g 1 ( w ) , . . .
, g m ( w )) across iterations, where g ( w ) has m blocks and it is only allowed to probe O (1) blocks to attain their stochastic values and Jacobians.
To improve the complexity for solving these problems, we propose a novel stochastic method named Multi-block-Singleprobe Variance Reduced (MSVR) estimator to track the sequence of g ( w ) .
It is inspired by STORM but introduces a customized error correction term to alleviate the noise not only in stochastic samples for the selected blocks but also in those blocks that are not sampled.
With the help of the MSVR estimator, we develop several algorithms for solving the aforementioned compositional problems with improved complexities across a spectrum of settings with non-convex/convex/strongly convex/Polyak-Łojasiewicz (PL) objectives.
Our results improve upon prior ones in several aspects, including the order of sample complexities and dependence on the strong convexity parameter.
Empirical studies on multi-task deep AUC maximization demonstrate the better performance of using the new estimator
Introduction
This paper is motivated by solving the following Finite-sum Coupled Compositional Optimization (FCCO) problem that has broad applications in machine learning [Wang and Yang, 2022]:
[Equation]
where f i : R p (cid:55)→ R is a simple deterministic function.
We assume that only noisy estimations of g i ( · ) and its Jacobian ∇ g i ( · ) can be accessed, denoted as g i ( · ; ξ i ) and ∇ g i ( · ; ξ i ) , where ξ i represents the random sample(s) drawn from a stochastic oracle such that E [ g i ( · ; ξ i )]
= g i ( · ) and
E [ ∇ g i ( · ; ξ i )]
= ∇ g i ( · ) .
A special case to be considered separately is when each ξ i has a finite support and is uniformly distributed.
In this case, the problem can be represented as:
[Equation]
These problems are different from classical stochastic compositional optimization (SCO) problems
E ζ [ f ζ ( E
[Equation]
g ( w ; ξ ))] and its finite-sum variant 1 /m (cid:80)
[Equation]
[Wang
[Equation]
], because the inner function is coupled with the outer index in FCCO.
A striking difference in solving FCCO problems is that we need to deal with multiple functional mappings of g i ( w ) for i = 1 , . . .
, m .
A challenge emerges when it is not possible to draw data samples for all blocks i = 1 , . . .
, m at each iteration due to some restrictions (e.g., limited memory and computational budget per-iteration).
Wang and Yang [2022] studied this problem comprehensively and proposed an algorithm named as SOX.
A key to their algorithmic design is to maintain and selectively update a sequence of estimators u
= ( u 1 , . . .
, u m ) for tracking g ( w ) = ( g 1 ( w ) , . . .
, g m ( w )) by exponential moving average, i.e.,
[Equation]
[Equation]
where ξ it and B 1 t ⊆ { 1 , . . .
, m } denote a set of sampled blocks.
With u , the gradient estimator is computed by exponential moving average as well.
As a result, they establish a sample complexity of O ( mϵ − 4 ) for non-convex objectives, O ( mϵ − 3 ) for convex objectives and O ( mµ − 2 ϵ − 1 ) for µstrongly convex objectives.
However, there are several caveats of these results: (i) the sample complexities (e.g., O ( mϵ − 4 ) for a non-convex objective) are no better than probing all blocks at each iteration, for which Ghadimi et al.
[2020] have established an O ( ϵ − 4 ) iteration complexity and an O ( mϵ − 4 ) sample complexity; (ii) when m = |B 1 t | = 1 , the problem reduces to a special case of classic SCO problems; however, the complexities are worse than the state-of-the-art (SOTA) sample complexities for non-convex, convex and strongly convex objectives, which are O
( ϵ − 3 ) , O ( ϵ − 2 ) and
O ( µ − 1 ϵ − 1 ) , respectively [Zhang and Xiao, 2019, Jiang et al., 2022].
A useful technique for achieving these complexities in prior works is by using variance reduction techniques, so a straightforward approach is to change the update of u it by using a variance reduced estimator and do similarly for the gradient estimator.
In particular, one can change the update for u it according to STORM [Cutkosky and Orabona, 2019]:
[Equation]
However, this simple change does not improve the complexities over that obtained by Wang and Yang [2022].
The reason is that the standard error correction term marked above in STORM only accounts for the randomness in g i ( w t ; ξ i t ) but not in the randomness caused by sampling i ∈ B t 1 .
So, a major question remains:
How can we further improve the complexities for solving FCCO to match the SOTA results of
SCO by using variance reduction techniques via probing only O (1) blocks at each iteration?
To address this issue, we propose a novel variance reduction technique by selectively updating u it for tracking g ( w t ) , to which we refer as Multi-block-Single-probe variance-reduced (MSVR) estimator.
It employs a similar update as STORM for selected u it but with a different customized error correction term to deal with the randomness in both g i ( w t ; ξ it ) and that in B t 1 .
Based on MSVR, we develop several algorithms for FCCO problems with different ways to compute the gradients, and analyze the sample complexities across a spectrum of settings with non-convex/convex/strongly convex/PL objectives and finite/infinite support of ξ i .
We summarize our contributions and our results below:
• We develop a novel MSVR estimator for tracking a sequence of multiple blocks of functional mappings by only probing O (1) blocks via random samples at each iteration.
[Table]
[Table]
• By applying the MSVR estimator, we develop three algorithms for FCCO by using different methods for computing the gradients, and establish improved complexities for non-convex, convex, strongly convex, and PL objectives.
A comparison between our algorithms and existing methods is shown in Table 1, where we also exhibit the dependence on B 2 , which is the size of the inner batch for estimating each g
i ( w ) .
• The complexity of our first method (i.e., MSVR-v1) enjoys the same order on ϵ as SOX, but does not depend on m ; MSVR-v2 improves the dependence on ϵ , and its complexities match the SOTA results for SCO when m = 1 ; our MSVR-v3 further reduces the dependence on ϵ for the finite support of ξ , and also attains the SOTA complexities when m = 1 .
• We conduct experiments on multi-task deep AUC maximization to verify the theory and demonstrate the advantage of the proposed algorithms.
Related work
This section briefly reviews related work on variance-reduced methods and stochastic compositional optimization (SCO) problems.
Variance-reduction (VR) techniques for improving the convergence of stochastic optimization originate from Roux et al.
[2012] for solving convex finite-sum empirical risk minimization (ERM) problems.
Since then, different VR techniques have been proposed for convex finite-sum ERM, e.g.,
SVRG [Johnson and Zhang, 2013, Zhang et al., 2013] and SAGA [Defazio et al., 2014].
These works have improved the complexity for solving smooth and strongly convex problems to a logarithmic complexity.
For non-convex ERM problems, Fang et al.
[2018] invents the SPIDER estimator similar to its predecessor SARAH
[Nguyen et al., 2017], and improve the complexity of standard SGD from
O ( ϵ − 4 ) to O ( ϵ − 3 ) and O ( √ nϵ − 2 ) in stochastic and finite-sum settings, respectively, where n is the number of components in the finite-sum.
Algorithmic improvements have been made to SPIDER by using a constant step size in SpiderBoost [Wang et al., 2018] and using a constant batch size in
STORM [Cutkosky and Orabona, 2019].
Several classes of SCO have been studied.
The first class is the two-level SCO whose objective is given by E ξ [ f ξ ( E ω [ g ω ( w )])] , where ξ and ω are random variables.
While the study of two-level compositional functions dates back to the 70s, the most recent comprehensive study was initiated by
Wang et al.
They proposed a two time-scale classic algorithm named SCGD and establish its asymptotic guarantee and non-asymptotic convergence rates.
Following this work, many studies have been devoted to improving the convergence rates or algorithmic design of two-level SCO [Wang et al., 2016, Ghadimi et al., 2020, Zhang and Lan, 2021].
In particular, recent works have used variance-reduction techniques based on SPIDER/SARAH/STORM to estimate the inner values and the gradients [Liu et al., 2018, Yuan et al., 2019a, Zhang and Xiao, 2019, Chen et al., 2021, Qi et al., 2021a].
Similar efforts have been extended to the second class of SCO, i.e., multi-level SCO with an objective E ξ 1 [ f 1 ξ 1 ( E ξ 2 [ f 2 ξ 2 ( . . .
( E ξ K [ f Kξ K ( w ))] . . . )])]
[Yang et al., 2019].
Recent studies have been focused on further improving the sample complexity and reducing the dependence on the number of levels K [Balasubramanian et al., 2021, Chen et al., 2021, Zhang and Lan, 2021, Zhang and Xiao, 2021, Jiang et al., 2022].
These works also employed variance reduction techniques to design their own methods.
However, directly applying these algorithms of two-level and multi-level SCO to
FCCO requires probing all
m blocks in g ( w ) , which is prohibitive in many applications.
The third class of SCO is the Conditional Stochastic Optimization (CSO) whose objective is in the form of E ξ [ f ξ ( E ω | ξ g ω ( w ; ξ )])]
[Hu et al., 2020], where ω | ξ means that the distribution of ω might depend on ξ .
The FCCO problem can be considered as a special case of CSO.
The key difference from the first class of SCO discussed above is that the inner function g depends on the random variable ξ of the outer level.
For CSO, Hu et al.
[2020] proposed two algorithms with and without using the variance-reduction technique (SpiderBoost) named BSGD and BSpiderboost, and established complexities for non-convex, convex and strongly convex functions, which are shown in Table 1.
However, their algorithms require a large batch size for estimating the inner functions.
Recently, a novel class (the fourth class) of SCO was studied, which is referred to as the finite-sum coupled compositional optimization (FCCO)
[Wang and Yang, 2022].
The finite-sum structure makes it possible to develop more practical algorithms without relying on huge batch size per-iteration.
It was first studied by Qi et al.
[2021b] for maximizing the point-estimator of the area under the precision-recall curve.
Recently, it was comprehensively investigated by Wang and Yang [2022] and more applications of FCCO have been demonstrated in machine learning.
Nevertheless, their algorithm—SOX does not use variance reduction techniques and hence suffers from the limitations discussed in the previous section.
Proposed Algorithms and Convergence
First, we introduce the notations and assumptions used in this paper.
Then we describe the MSVR estimator in detail and develop algorithms based on the proposed estimator.
3.1 Notations and Assumptions
Let [ m ] = { 1 , . . .
, m } .
The definition of sample complexity is given below, which is widely used to measure the efficiency of stochastic algorithms.
Definition
The sample complexity is the number of samples needed to find a point satisfying E [ ∥∇ F ( w ) ∥ ] ≤ ϵ ( ϵ -stationary) or E [ F ( w ) − inf w F ( w )] ≤ ϵ ( ϵ -optimal).
Next, we make following assumptions throughout the paper, which are commonly used in the studies of SCO [Wang et al., 2016, 2017, Yuan et al., 2019a, Zhang and Xiao, 2019, 2021].
Assumption 1 (Smoothness and Lipschitz continuity)
We assume that each f i is L f -smooth and
C f -Lipchitz continuous, each g i is L g -smooth and C g -Lipschitz continuous.
Remark: This implies F ( w ) is C F -Lipchitz continuous and L F -smooth, where C F = C f C g ,
L F = C
f L g + C 2 g L f
[Zhang and Xiao, 2021].
Assumption 2 (Bounded variance)
[Equation]
where the random variable ξ it denotes a batch of samples with batch size B 2 ≥ 1 .
Assumption 3 (Average Lipchitz continuity of g i and its Jacobian)
[Equation]
Remark:
Although this assumption seems strong at the first sight, it is quite standard and widely used in the recent compositional optimization literature [Yuan et al., 2019a, Zhang and Xiao, 2019, 2021, Jiang et al., 2022].
Assumption 4 F ∗ = inf w F ( w ) ≥ −∞ and F ( w 1 ) − F ∗ ≤ ∆ F for the initial solution w 1 .
3.2 Multi-block-Single-probe Variance Reduced (MSVR) Estimator
Assume that we have a budget to probe only B 1 out of m mappings in g ( w ) .
To this end, at the t -th iteration we sample a set of blocks B t 1 ⊆
[ m ] , where |B t 1 | =
B 1 , and probe the corresponding g i ( w ) by accessing the noisy estimates g i ( w t ; ξ it ) for i ∈ B t 1 .
Then, we just update the corresponding block in our estimator u t .
Specifically, we update u it for i ∈ B t 1 in a new way and keep other blocks unchanged.
The whole estimator is shown below:
[Equation]
The first line of our estimator is inspired by STORM [Cutkosky and Orabona, 2019].
The difference is that the STORM estimator sets γ t =
(1 − β t ) , while for MSVR, γ t is set as m − B 1 B 1 (1 − β t )
+ (1 − β t ) according to our analysis.
We name equation (5) as Multi-block-Single-probe Variance Reduced (MSVR) estimator.
By multi-block, we mean the estimator can track multiple functional mappings ( g 1 , g 2 , · · · , g m ) , simultaneously; by single-probe, we indicate the number of sampled blocks Bfor probing can be as small as one.
It is notable that when B 1 = m , i.e., all blocks are probed at each iteration, γ t = 1 − β t and MSVR reduces to STORM applied to g ( w ) .
The additional factor in γ t , i.e., γ 0 t = m − B 1 B 1 (1 − β t ) is to account for the randomness in the sampled blocks and noise in those blocks that are not updated.
To briefly understand the additional factor γ 0 t , we consider bounding ∥ u t − g ( w t ) ∥ 2 = (cid:80) mi =1 ∥ u it − g
i ( w t ) ∥ 2 .
Let us focus on a fixed i ∈ [ m ] .
Then we have
E
[Equation]
Note that the first term A 1 in the R.H.S. can be bounded similarly as STORM by building recurrence with ∥ u it − 1 − g i ( w t − 1 ) ∥ 2 .
However, there exists the second term due to the randomness of B t 1 , which can be decomposed as
[Equation]
The first two terms in R.H.S. ( A 21 and A 22 ) can be easily handled.
The difficulty comes from the third term, which cannot be simply bounded by using Young’s inequality.
If doing so, it will end up with a non-diminishing error of u it .
To combat this difficulty, we use the additional factor brought by γ 0 t ( g i (cid:0) w t ; ξ it (cid:1)
− g i (cid:0) w t − 1 ; ξ it (cid:1) ) in A 1 to cancel A 23 .
This is more clear by the following decomposition of A 1 .
[Equation]
In light of the above decomposition, we can bound E [ ∥ A 11 + A 12 ∥ 2 ] ≤ E [ ∥
A 11 ∥ 2 + ∥
A 12 ∥ 2 + 2 A ⊤ 11 A 12 ] and E [ ∥ A 13 + A 14 ∥ 2 ] ≤ 2 E [ ∥
A 13 ∥ 2 ] + 2 E [ ∥ A 14 ∥ 2 ] .
The resulting term E [2 A ⊤ 11 A 12 ] has a negative sign as A 23 .
Hence, by carefully choosing γ 0 t , we can cancel both terms.
The remaining terms can be organized similarly as in the analysis for STORM.
We give a technical lemma for building the recurrence of MSVR’s error below.
All the proofs are deferred to the supplementary material due to space limitations.
Lemma
By setting γ t = m − B 1 B 1 (1 − β t ) + (1 − β t ) , for β t ≤ 1 2 , we have:
[Equation]
1: Input do 3:
Compute estimator u to equation
Compute
STORM
8: end
9: Choose
10: Return
Remark:
The above recursion is similar to that of STORM for tracking a sequence of a single-block functional mapping.
Since the last term ∥ w t − w t − 1 ∥ 2 can be offset in the future analysis, intuitively the estimation error ∥
u t − g ( w t ) ∥ 2 would reduce after each iteration.
Single Point Version.
A limitation of the MSVR estimator is that it needs to probe selected blocks at two different points, i.e., g i ( w t ; ξ it ) and g i ( w t − 1 ; ξ it ) .
With a more careful analysis, we can probe a selected block at a single point similar to that used by Balasubramanian et al.
[2021] and Chen et al.
Specifically, we replace g i (cid:0) w t ; ξ it (cid:1)
− g i (cid:0) w t − 1 ; ξ it (cid:1) with ∇ g i (cid:0) w t ; ξ it (cid:1) ⊤ ( w t − w t − 1 ) .
As a result, we propose a single-point version of MSVR (named as MSVR-SP) estimator below:
[Equation]
The MSVR-SP estimator enjoys the similar recurrence for the estimation error.
Lemma 2 Set γ t = m − B 1 B 1 (1 − β t )
+ (1 − β t ) .
If ∥ w t +1 − w t ∥ 2 ≤ η 2 t C 2 F and η t ≤ √ β t , we have:
[Equation]
Remark:
If there is a constraint on the range of g i , we can add a projection to the update of u it such that it always resides in the range, which will not affect the analysis of Lemma 1 and Lemma 2.
3.3 Leveraging the MSVR Estimator for solving the FCCO Problem
Now, we are ready to present our proposed algorithms for solving problem (1).
The first two algorithms (named MSVR-v1 and MSVR-v2) are presented in Algorithm 1.
These two methods differ in how to estimate the gradient.
Let us first consider MSVR-v1.
At each time step t , we first use the proposed MSVR or MSVR-SP estimator u t to estimate the inner function value.
Then, following the previous literature [Wang et al., 2021, Wang and Yang, 2022], we use the moving average estimator z t to estimate the gradient as:
[Equation]
where Π C F denotes the projection onto the ball with radius C F .
This projection is optional for using
MSVR, but is required for using MSVR-SP to ensure ∥ w t +1 − w t ∥ 2 ≤ η 2 t C 2 F as used in Lemma 2.
Since the true gradient ∇ F is also in this ball, i.e., ∥∇ F ∥ ≤ C F , the projection will not affect the future analysis.
Also note that when computing the estimator z t , we use ∇ f i ( u it − 1 ) instead of ∇ f i ( u it ) to avoid the dependence on the random variable ξ it , which may lead to dependent issues otherwise.
Finally, we use the estimated gradient z t to update the parameter w t +1 .
Now, we provide the theoretical guarantee for the MSVR-v1 method.
Theorem 1 Our MSVR-v1 algorithm with α t +
=
O ( η t ) , β t +
=
O ( m 2 η 2 t B 21 ) , a = O ( mB 2 B 1 ) and η
[Equation]
= min (cid:26)(cid:16)
[Equation]
, can find an ϵ -stationary point in O
[Equation]
Remark:
This complexity is strictly better than previous SOTA method SOX, which enjoys an iteration complexity of O (cid:16) max (cid:110)
[Equation]
The sample complexity can be obtained by multiplying the iteration complexity with B 1 B 2 .
We can see that larger B 1 or B 2 yields a smaller iteration complexity, which means that from the computational perspective, if samples can be processed in parallel (e.g., in GPU), there is a benefit of using large B 1 and/or B 2 .
However, from the sample complexity perspective, using B 1 = B 2 = 1 is the best.
The same discussion holds for other theorems below.
However, the complexity of MSVR-v1 is still on the order of O ( ϵ − 4 ) .
Due to the biased nature of the estimated gradient, using the moving average update is not enough for achieving the SOTA complexity of O ( ϵ − 3 ) .
So, we use the technique of STORM [Cutkosky and Orabona, 2019] to update z t as follows:
[Equation]
where the projection operation is needed if using the MSVR estimator.
Now, we prove this new method (i.e., MSVR-v2) can obtain the optimal complexity of O ( ϵ − 3 ) .
Theorem 2 Our MSVR-v2 algorithm with α t +
= O ( mη 2 t B 1 ) ,
β t +1 =
O (cid:16)
m 2 η 2 t B 21 (cid:17) , a = O ( mB 2 B 1 ) and η t = O (cid:16) ( B 1 √ B 2 m )
[Equation]
/ 3 ( a + t ) −
[Equation]
3 (cid:17) , can find an ϵ -stationary point in O (cid:16) mϵ − 3 B 1 √ B 2 (cid:17) iterations.
Remark: When m = 1 and f is the identity function, problem (1) reduces to the standard stochastic non-convex optimization, whose lower bound is Ω (cid:0) ϵ − 3 (cid:1)
[Arjevani et al., 2019], indicating our
MSVR-v2 is optimal.
Next, we show that the complexity can be further improved when the objective function is convex or strongly convex.
We note that Polyak-Łojasiewicz (PL) [Karimi et al., 2016] objectives are more general than strongly convex functions, since µ -strong convexity implies the µ -PL condition.
So, we will consider the PL condition and introduce its definition below.
Definition 2 F ( w ) satisfies the µ -PL condition if there exists µ > 0 such that:
[Equation]
Then, we derive improved rates for convex or PL objectives by using the stage-wise design given in
Algorithm 3 in the supplement.
Theorem
If the objective function satisfies the convexity or µ -PL condition, MSVR-v1 derives a sample complexity of O (max( B 1 , B 2 )
ϵ − 3 ) or O (max( B 1 , B 2 ) µ − 2 ϵ − 1 ) , separately.
For MSVR-v2, the complexity can be further improved to O (cid:0) m √ B 2 ϵ − 2 (cid:1) or O (cid:0) m √ B 2 µ − 1 ϵ − 1 (cid:1) .
Remark: The complexities for MSVR-v2 are optimal, since they match the Ω (cid:0) ϵ − 2 (cid:1) and Ω (cid:0) µ − 1 ϵ − 1 (cid:1)
lower bound for stochastic convex and strongly convex optimization [Agarwal et al., 2012].
Remark:
The algorithms proposed in this paper can also use adaptive (Adam-style) learning rates and obtain the same complexity using the techniques proposed by Guo et al.
The details are provided in the supplementary.
Sample a
Compute function value estimator u t according equation
Compute gradient z t
11: end
12: Choose uniformly
13: Return w τ
An Improved Rate for the Finite-sum Case
In this section, we consider the case that inner function
g
i is in the form of the finite-sum, i.e., g i ( w ) = 1 n (cid:80) nj =
g i ( w ; ξ ij ) , so that we can compute the exact value of g i ( w ) in some iterations.
We first modify our MSVR estimator to utilize the finite-sum structure.
Inspired by SVRG [Johnson and
Zhang, 2013, Zhang et al., 2013], we compute a full version of the inner function value for every I iterations at w τ , i.e., g i ( w τ ) =
1 n (cid:80) nj =
g
i ( w τ ; ξ ij ) for i = 1 , · · · , m , where τ mod I = 0 .
Then, in each step, we use
[Equation]
to replace g i ( w t ; ξ it ) in the origin estimator.
In this way, our MSVR estimator is changed to:
[Equation]
For this estimator, we have the following guarantee.
Lemma 3 If β ≤ 12 and βI ≤ mB 1 , by setting γ = m − B 1 B 1 (1 − β ) + (1 − β ) , we have:
[Equation]
Remark:
Compared with Lemma 1, we remove the 2 B 1 β 2 σ 2 Bterm, which is the key to reduce the complexity since we can now use a larger parameter β .
To attain the optimal complexity, we modify the gradient estimator z t in a similar way:
[Equation]
where h t involves both the full gradient and the stochastic gradient (we also need to save each ∇ f i ( u τ − 1 ) and calculate the full version of 1 m (cid:80) mi =1 ∇ f i ( u iτ − 1 ) ∇ g
i ( w τ ) at those steps τ )
[Equation]
The whole method is summarized in Algorithm 2 (named as MSVR-v3).
Next, we show that
MSVR-v3 is equipped with an optimal complexity of O ( √ nϵ − 2 ) .
Theorem 4 Our MSVR-v3 with I = mnB 1 B 2 , α = O (cid:0) B 1 B 2 mn (cid:1) , β
= O (cid:0) B 2 n (cid:1) and η = O (cid:16) B 1 √ B 2 m √ n (cid:17) , can obtain an ϵ -stationary point in T
= O (cid:16) m √ nϵ − 2 B 1 √ B 2 (cid:17) iterations.
[Figure]
Remark: When m = 1 and f is the identity function, problem (2) reduces to the stochastic finite-sum optimization, whose optimal complexity is O (cid:0) √ nϵ − 2 (cid:1)
[Fang et al., 2018, Li et al., 2021], indicating our complexity is optimal in terms of ϵ and n .
Similarly, a better complexity can be obtained under the convexity or PL condition.
Theorem
If the objective function satisfies the convexity or µ -PL condition, the sample complexity can be improved to O (cid:16) m √ nϵ − 1 B 1 √ B 2 log 1 ϵ (cid:17) or O (cid:16) m √ nµ − 1 B 1 √ B 2 log
1 ϵ (cid:17) , respectively.
Remark: It is notable that we achieve a linear convergence rate O (cid:0) log 1 ϵ (cid:1) under the PL condition, matching the current result in the single-level finite-sum problem [Li et al., 2021]
Experiments
In this section, we conduct experiments on the multi-task deep AUC maximization to evaluate the proposed methods and we will consider more applications in the long version of the paper.
For binary classification (label y = 1 or y = − 1 ), AUC maximization can be formulated as minimizing the following composite loss [Zhu et al., 2022]:
[Equation]
where a ( w ) =
E [ h w ( x ) | y = 1] , b ( w ) =
E [ h w ( x ) | y = − 1] and ℓ ( · ) is a surrogate function.
The above objective recovers the pairwise square loss and the min-max margin loss proposed by Yuan et al.
[2020] for deep AUC maximization by setting ℓ ( · ) as the square function or squared hinge function, respectively.
When applied to multi-task classification (e.g., multiple classes), we can optimize the averaged AUC losses over all tasks, i.e., AUC = 1 m (cid:80) mi =1 AUC ( i ) .
The nested structure only comes from the term ℓ ( a ( w ) − b ( w )) , and we can rewrite it as the form of FCCO problem, where
[Equation]
where D i + / − denots the positive/negative datasets of the i -th task.
Configurations.
In the experiment, we follow the setup in Zhu et al.
[2022] and set the surrogate function ℓ as squared hinge ℓ ( x ) =
12 (max { c + x, 0 } ) 2 .
We use ResNet18 as backbone network, and train on six datasets:
STL10 [Coates et al., 2011], CIFAR10 [Krizhevsky, 2009], CIFAR100 [Krizhevsky, 2009], MNIST [LeCun et al., 1998], Fashion-MNIST [Xiao et al., 2017], and SVHN [Netzer et al., 2011].
We compare our methods with previous SOTA algorithm SOX
[Wang and Yang, 2022].
For our methods, parameters α and β are searched from { 0 . 1 , 0 . 5 , 0 . 9 , 1 . 0 } .
For SOX algorithm, its parameters β and γ are searched from the same set.
B 1 is set as 50 for CIFAR100 and 5 for other datasets.
Inner batch size B 2 is chosen as 128 for all methods.
We tune the learning rate from the set { 1 e − 4 , 1 e − 3 , 2 e − 3 , 5 e − 3 , 1 e − 2 } and pick the best one for each method.
The experiments are conducted on single NVIDIA Tesla M40 GPU.
Results.
Figure 1 shows the loss against the number of samples drawn by different methods, and all curves are averaged over 5 runs.
We observe that MSVR-V1 is better than SOX on the CIFAR dataset, and close to it on other datasets.
MSVR-v2 converges faster than SOX and MSVR-v1, and the loss of MSVR-v3 decreases most rapidly, demonstrating a low sample complexity.
Conclusion and Future Work
In this paper, we develop a novel MSVR estimator for tracking multiple functional mappings by probing only O (1) blocks.
Equipped with this estimator, we design three algorithms for FCCO problems and obtain improved complexities across a spectrum of settings.
Experimental results on multi-task deep AUC maximization also verify the effectiveness of our methods.
In future work, we will investigate other applications that can be solved by using the proposed estimator.
Acknowledgments and Disclosure of Funding
W. Jiang, Y. Wang and L. Zhang were partially supported by NSFC (62122037, 61921006).
G. Li and T. Yang were partially supported by Amazon research award.
The authors would like to thank the anonymous reviewers for their helpful comment.
[-Solve γ t +
-]
[-and we have γ t +
-]
[-Lemma 
-]
[-u t − g ( w t ) ∥ 
-]
[-By setting β t +1 = 
-]
[-m 2 C 2 η 2 t B(and note that c 0 = 72 C , α t +1 = 
-]
[-denote M = ∆ F + 2 C c 
-]
[-Denote M = ∆ F + 18 η 0 + c 40 
-]
[-(cid:13)(cid:13) 
-]
[-O (cid:16) m √ n B 1 √ B 2 ϵ 2 (cid:17) and T 
-]
[-Theorem 6 If we choose parameters α t +
-]