Multi-block-Single-probe Variance Reduced Estimator for Coupled Compositional Optimization
Anonymous Author(s)
AffiliationAddress email
Abstract
Variance reduction techniques such as SPIDER/SARAH/STORM have been extensively studied to improve the convergence rates of stochastic non-convex optimization, which usually maintain and update a sequence of estimators for asingle function across iterations.
What if we need to track multiple functionalmappings across iterations but only with access to stochastic samples of O (1)functional mappings at each iteration?
There is an important application in solvingan emerging family of coupled compositional optimization problems in the formof (cid:80) mi =
f
i
g i ( w )) , where g i is accessible through a stochastic oracle.
The keyissue is to track and estimate a sequence of g ( w )
=
( g 1 ( w ) , . . .
, g m ( w )) acrossiterations, where g ( w ) has m blocks and it is only allowed to probe O (1) blocks toattain their stochastic values and Jacobians.
To improve the complexity for solvingthese problems, we propose a novel stochastic method named Multi-block-Singleprobe Variance Reduced (MSVR) estimator to track the sequence of g ( w ) .
It isinspired by STORM but introduces a customized error correction term to alleviatethe noise not only in stochastic samples for the selected blocks but also in thoseblocks that are not sampled.
With the help of the MSVR estimator, we developseveral algorithms for solving the aforementioned compositional problems with improved complexities across a spectrum of settings with non-convex/convex/stronglyconvex/Polyak-Łojasiewicz (PL) objectives.
Our results improve upon prior onesin several aspects, including the order of sample complexities and dependenceon the strong convexity parameter.
Empirical studies on multi-task deep AUCmaximization demonstrate the better performance of using the new estimator.
Introduction
This paper is motivated by solving the following Finite-sum Coupled Compositional Optimization(FCCO) problem that has broad applications in machine learning [Wang and Yang, 2022]
[Equation]
where f i : R p (cid:55)→ R is a simple deterministic function.
We assume that only noisy estimationsof g i ( · ) and its Jacobian ∇ g i ( · ) can be accessed, denoted as g i ( · ; ξ i ) and ∇ g i ( · ; ξ i ) , where ξ i 27 represents the random sample(s) drawn from a stochastic oracle such that E [ g i ( · ; ξ i )]
= g i ( · ) and 28 E [ ∇ g i ( · ; ξ i )]
= ∇ g i ( · ) .
A special case to be considered separately is when each ξ i has a finitesupport and is uniformly distributed.
In this case, the problem can be represented as:
[Equation]
These problems are different from classical stochastic compositional optimization (SCO) problems 31 E ζ [ f ζ ( E ξ g ( w ; ξ ))] and its finite-sum variant 1 /m (cid:80) mi
=
f i (1 /n (cid:80) nj =1 g ( w ; ξ j ))
[Wang et al., 2017],because the inner function is coupled with the outer index in FCCO.
A striking difference in solving FCCO problems is that we need to deal with multiple functionalmappings of g i ( w ) for i = 1 , . . .
, m .
A challenge emerges when it is not possible to draw data samples for all blocks i = 1 , . . .
, m at each iteration due to some restrictions (e.g., limited memory andcomputational budget per-iteration).
Wang and Yang [2022] studied this problem comprehensivelyand proposed an algorithm named as SOX.
A key to their algorithmic design is to maintain and selectively update a sequence of estimators u
= ( u 1 , . . .
, u m ) for tracking g ( w ) = ( g 1 ( w ) , . . .
, g m ( w ))by exponential moving average, i.e.,
[Equation]
where ξ it and B 1 t ⊆ { 1 , . . .
, m } denote a set of sampled blocks.
With u , the gradient estimator iscomputed by exponential moving average as well.
As a result, they establish a sample complexityof O
( mϵ − 4 ) for non-convex objectives, O ( mϵ − 3 ) for convex objectives and O ( mµ − 2 ϵ − 1 ) for µstrongly convex objectives.
However, there are several caveats of these results: (i) the samplecomplexities (e.g., O ( mϵ − 4 ) for a non-convex objective) are no better than probing all blocks ateach iteration, for which Ghadimi et al.
[2020] have established an O ( ϵ − 4 ) iteration complexity andan O ( mϵ − 4 ) sample complexity; (ii) when m = |B 1 t | = 1 , the problem reduces to a special case ofclassic SCO problems; however, the complexities are worse than the state-of-the-art (SOTA) samplecomplexities for non-convex, convex and strongly convex objectives, which are O
( ϵ − 3 ) , O ( ϵ − 2 ) and
O ( µ − 1 ϵ − 1 ) , respectively [Zhang and Xiao, 2019, Jiang et al., 2022].
A useful technique for achievingthese complexities in prior works is by using variance reduction techniques, so a straightforwardapproach is to change the update of u it by using a variance reduced estimator and do similarly for thegradient estimator.
In particular, one can change the update for u it according to STORM [Cutkoskyand Orabona, 2019]:
[Equation]
However, this simple change does not improve the complexities over that obtained by Wang and Yang[2022].
The reason is that the standard error correction term marked above in STORM only accountsfor the randomness in g i ( w t ; ξ it ) but not in the randomness caused by sampling i ∈ B t 1 .
So, a majorquestion remains:
How can we further improve the complexities for solving FCCO to match the SOTA results of
SCO by using variance reduction techniques via probing only O (1) blocks at each iteration?
To address this issue, we propose a novel variance reduction technique by selectively updating u it fortracking g ( w t ) , to which we refer as Multi-block-Single-probe variance-reduced (MSVR) estimator.
It employs a similar update as STORM for selected u it but with a different customized errorcorrection term to deal with the randomness in both g i ( w t ; ξ it ) and that in B t 1 .
Based on MSVR,we develop several algorithms for FCCO problems with different ways to compute the gradients,and analyze the sample complexities across a spectrum of settings with non-convex/convex/stronglyconvex/PL objectives and finite/infinite support of ξ i .
We summarize our contributions:
• We develop a novel MSVR estimator for tracking a sequence of multiple blocks of functionalmappings by only probing O (1) blocks via random samples at each iteration.• By applying the MSVR estimator, we develop three algorithms for FCCO by using differentmethods for computing the gradients, and establish improved complexities for non-convex,convex, strongly convex, and PL objectives.
A comparison between our algorithms andexisting methods is shown in Table 1, where we also exhibit the dependence on B 2 , whichis the size of the inner batch for estimating each g i ( w ) .•
The complexity of our first method (i
e., MSVR-v1) enjoys the same order on ϵ as SOX, butdoes not depend on m ; MSVR-v2 improves the dependence on ϵ , and its complexities matchthe SOTA results for SCO when m = 1 ; our MSVR-v3 further reduces the dependence on ϵfor the finite support of ξ , and also attains the SOTA complexities when m = 1 .
[Table]
[Table]
• We conduct experiments on multi-task deep AUC maximization to verify the theory and [Table] demonstrate the advantage of the proposed algorithms [Table]
Related work
[Table]
This section briefly reviews related work on variance-reduced methods and stochastic compositionaloptimization (SCO) problems.
Variance-reduction (VR) techniques for improving the convergence of stochastic optimization originate from Roux et al.
[2012] for solving convex finite-sum empirical risk minimization (ERM)problems.
Since then, different VR techniques have been proposed for convex finite-sum ERM, e.g.,
SVRG [Johnson and Zhang, 2013, Zhang et al., 2013] and SAGA [Defazio et al., 2014].
These workshave improved the complexity for solving smooth and strongly convex problems to a logarithmiccomplexity.
For non-convex ERM problems, Fang et al.
[2018] invents the SPIDER estimator similarto its predecessor SARAH
[Nguyen et al., 2017], and improve the complexity of standard SGD from
O ( ϵ − 4 ) to O ( ϵ − 3 ) and O ( √ nϵ − 2 ) in stochastic and finite-sum settings, respectively, where n is thenumber of components in the finite-sum.
Algorithmic improvements have been made to SPIDERby using a constant step size in SpiderBoost [Wang et al., 2018] and using a constant batch size in
STORM [Cutkosky and Orabona, 2019].
Several classes of SCO have been studied.
The first class is the two-level SCO whose objective isgiven by E ξ
[ f ξ ( E ω [ g ω ( w )])] , where ξ and ω are random variables.
While the study of two-levelcompositional functions dates back to the 70s, the most recent comprehensive study was initiated by
Wang et al.
They proposed a two time-scale classic algorithm named SCGD and establishits asymptotic guarantee and non-asymptotic convergence rates.
Following this work, many studieshave been devoted to improving the convergence rates or algorithmic design of two-level SCO [Wanget al., 2016, Ghadimi et al., 2020, Zhang and Lan, 2021].
In particular, recent works have usedvariance-reduction techniques based on SPIDER/SARAH/STORM to estimate the inner values andthe gradients [Liu et al., 2018, Yuan et al., 2019a, Zhang and Xiao, 2019, Chen et al., 2021, Qi et al.
,2021a].
Similar efforts have been extended to the second class of SCO, i.e., multi-level SCO with anobjective E ξ 1 [ f 1 ξ 1 ( E ξ 2 [ f 2 ξ 2 ( . . .
( E ξ K [ f Kξ K ( w ))] . . . )])]
[Yang et al., 2019].
Recent studies have beenfocused on further improving the sample complexity and reducing the dependence on the number oflevels K
[Balasubramanian et al., 2021, Chen et al., 2021, Zhang and Lan, 2021, Zhang and Xiao,2021, Jiang et al., 2022].
These works also employed variance reduction techniques to design theirown methods.
However, directly applying these algorithms of two-level and multi-level SCO to
FCCO requires probing all
m blocks in g ( w ) , which is prohibitive in many applications.
The third class of SCO is the Conditional Stochastic Optimization (CSO) whose objective is in theform of E ξ [ f ξ ( E ω | ξ g ω ( w ; ξ )])]
[Hu et al., 2020], where ω | ξ means that the distribution of ω mightdepend on ξ .
The FCCO problem can be considered as a special case of CSO.
The key difference from the first class of SCO discussed above is that the inner function g depends on the random variableξ of the outer level.
For CSO, Hu et al.
[2020] proposed two algorithms with and without usingthe variance-reduction technique (SpiderBoost) named BSGD and BSpiderboost, and establishedcomplexities for non-convex, convex and strongly convex functions, which are shown in Table 1.
However, their algorithms require a large batch size for estimating the inner functions.
Recently, a novel class (the fourth class) of SCO was studied, which is referred to as the finite-sumcoupled compositional optimization (FCCO)
[Wang and Yang, 2022].
The finite-sum structure makesit possible to develop more practical algorithms without relying on huge batch size per-iteration.
It was first studied by Qi et al.
[2021b] for maximizing the point-estimator of the area under theprecision-recall curve.
Recently, it was comprehensively investigated by Wang and Yang [2022]and more applications of FCCO have been demonstrated in machine learning.
Nevertheless, theiralgorithm—SOX does not use variance reduction techniques and hence suffers from the limitationsdiscussed in the previous section.
Proposed Algorithms and Convergence
First, we introduce the notations and assumptions used in this paper.
Then we describe the MSVRestimator in detail and develop algorithms based on the proposed estimator.
3.1 Notations and Assumptions
Let [ m ] = { 1 , . . .
, m } .
The definition of sample complexity is given below, which is widely used tomeasure the efficiency of stochastic algorithms.
Definition
The sample complexity is the number of samples needed to find a point satisfying 132 E [ ∥∇ F ( w ) ∥ ] ≤ ϵ ( ϵ -stationary) or E [ F ( w ) − inf w F ( w )] ≤ ϵ ( ϵ -optimal).
Next, we make following assumptions throughout the paper, which are commonly used in the studiesof SCO
[Wang et al., 2016, 2017, Yuan et al., 2019a, Zhang and Xiao, 2019, 2021].
Assumption 1 (Smoothness and Lipschitz continuity)
We assume that each f i is L f -smooth and
C f -Lipchitz continuous, each g i is L g -smooth and C g -Lipschitz continuous.
Remark: This implies F ( w ) is L F -smooth, where L F = C
f L g + C 2 g L f [Zhang and Xiao, 2021].
Assumption 2 (Bounded variance)
[Equation]
where the random variable ξ it denotes a batch of samples with batch size B 2 ≥ 1 .
Assumption 3 (Average Lipchitz continuity of g i and its Jacobian)
[Equation]
Assumption 4 F ∗ = inf w F ( w ) ≥ −∞ and F ( w 1 ) − F ∗ ≤ ∆ F for the initial solution w 1 .
3.2 Multi-block-Single-probe Variance Reduced (MSVR) Estimator
Assume that we have a budget to probe only B 1 out of m mappings in g ( w ) .
To this end, at thet -th iteration we sample a set of blocks B t 1 ⊆
[ m ] , where |B t 1 | =
B 1 , and probe the corresponding [-
-]145 g i ( w ) by accessing the noisy estimates g i ( w t ; ξ it ) for i ∈ B t 1 .
Then, we just update the correspondingblock in our estimator u t .
Specifically, we update u i t for i ∈ B t 1 in a new way and keep other blocksunchanged.
The whole estimator is shown below:
[Equation]
The first line of our estimator is inspired by STORM [Cutkosky and Orabona, 2019].
The differenceis that the STORM estimator sets γ t
= (1 − β t ) , while for MSVR, γ t is set as m − B 1 B 1 (1 − β t )
+ (1 − β t )according to our analysis.
We name equation (5) as Multi-block-Single-probe Variance Reduced(MSVR) estimator.
By multi-block, we mean the estimator can track multiple functional mappings( g 1 , g 2 , · · · , g m ) , simultaneously; by single-probe, we indicate the number of sampled blocks B 1for probing can be as small as one.
It is notable that when B 1 = m , i.e., all blocks are probed ateach iteration, γ t = 1 − β t and MSVR reduces to STORM applied to g ( w ) .
The additional factor inγ t , i.e., γ 0 t = m − B 1 B 1 (1 − β t ) is to account for the randomness in the sampled blocks and noise in thoseblocks that are not updated.
To briefly understand the additional factor γ 0 t , we consider bounding
[Equation]
−
[Equation]
w [Equation] ∥ 2 .
Let us focus on a fixed i ∈ [ m ] .
Then we have
[Equation]
Note that the first term A 1 in the R.H.S. can be bounded similarly as STORM by building recurrencewith ∥ u
it − 1 − g i ( w t − 1 )
∥ 2 .
However, there exists the second term due to the randomness of B t 1 , whichcan be decomposed as
[Equation]
The first two terms in R.H.S. ( A 21 and A 22 ) can be easily handled.
The difficulty comes fromthe third term, which cannot be simply bounded by using Young’s inequality.
If doing so, it willend up with a non-diminishing error of u it .
To combat this difficulty, we use the additional factorbrought by γ 0 t ( g i (cid:0) w t ; ξ it (cid:1)
− g i (cid:0) w t − 1 ; ξ it (cid:1) ) in A 1 to cancel A 23 .
This is more clear by the followingdecomposition of A 1 .
[Equation]
γ t ( g i ( w t ; ξ it ) − g i ( w t − 1 ; ξ it ) − g i ( w t )
+ g i ( w t − 1 ))
(cid:124) (cid:123)(cid:122) (cid:125) A∥ 2 ] ≤ E [ ∥ A 11 + A 12 ∥ 2 ] + E (cid:2) ∥
A 13 + A 14 ∥ 2 (cid:3) .
In light of the above decomposition, we can bound E [ ∥ A 11 + A 12 ∥ 2 ] ≤ E [ ∥
A 11 ∥ 2 + ∥
A 12 ∥ 2 +
A ⊤ 11 A 12 ] and E [ ∥ A 13 + A 14 ∥ 2 ] ≤ 2 E [ ∥
A 13 ∥ 2 ] + 2 E [ ∥ A 14 ∥ 2 ] .
The resulting term E [2 A ⊤ 11 A 12 ] hasa negative sign as A 23 .
Hence, by carefully choosing γ 0 t , we can cancel both terms.
The remainingterms can be organized similarly as in the analysis for STORM.
We give a technical lemma forbuilding the recurrence of MSVR’s error below.
All the proofs are deferred to the supplementarymaterial due to space limitations.
Lemma
By setting γ t = m − B 1 B 1 (1 − β t ) + (1 − β t ) , for β t ≤ 12 , we have:
[Equation]
Remark:
The above recursion is similar to that of STORM for tracking a sequence of a single-blockfunctional mapping.
Since the last term ∥ w t − w t − 1 ∥ 2 can be offset in the future analysis, intuitivelythe estimation error ∥
u t − g ( w t ) ∥ 2 would reduce after each iteration.
Single Point Version.
A of MSVR needs probe selected blocks at two different points, e. g i ( w t ; ξ it ) g i ( w t − 1 ; ξ it ) .
With a more careful analysis, we can probea block a point
Specifically, we replace g i (cid:0) w t ; ξ it (cid:1)
− g i (cid:0) w t − 1 ; ξ it (cid:1) with ∇ g i (cid:0) w t ; ξ it (cid:1) ⊤ ( w t − w t − 1 ) .
As aresult, we propose a single-point version of MSVR (named as MSVR-SP) estimator below:
[Equation]
The MSVR-SP estimator enjoys the similar recurrence for the estimation error.
Lemma 2 Set γ t = m − B 1 B 1 (1 − β t )
+ (1 − β t ) .
If ∥ w t +1 − w t ∥ 2 ≤ η 2 t L 2 F and η t ≤ √ β t , we have:
[Equation]
Remark:
If there is a constraint on the range of g i , we can add a projection to the update of u it suchthat it always resides in the range, which will not affect the analysis of Lemma 1 and Lemma 2.
3.3 Leveraging the MSVR Estimator for solving the FCCO Problem
Now, we are ready to present our proposed algorithms for solving problem (1).
The first twoalgorithms (named MSVR-v1 and MSVR-v2) are presented in Algorithm 1.
These two methodsdiffer in how to estimate the gradient.
Let us first consider MSVR-v1.
At each time step t , we first use the proposed MSVR or MSVR-SPestimator u t to estimate the inner function value.
Then, following the previous literature [Wang et al.,2021, Wang and Yang, 2022], we use the moving average estimator z t to estimate the gradient as:
[Equation]
where Π L F denotes the projection onto the ball with radius L F .
This projection is optional for using
MSVR, but is required for using MSVR-SP to ensure ∥ w t +1 − w t ∥ 2 ≤ η 2 t L 2 F as used in Lemma 2.
Since the true gradient ∇ F is also in this ball, i.e., ∥∇ F ∥ ≤ L F , the projection will not affect 195 the future analysis.
Also note that when computing the estimator z t , we use ∇ f i ( u it − 1 ) instead of∇
f i ( u it ) to avoid the dependence on the random variable ξ it , which may lead to dependent issuesotherwise.
Finally, we use the estimated gradient z t to update the parameter w t +1 .
Now, we providethe theoretical guarantee for the MSVR-v1 method.
Theorem 1 Our MSVR-v1 algorithm with α t +
=
O ( η t ) , β t +
=
O ( m 2 η 2 t B 21 ) , a = O ( mB 2 B 1 ) andη t = min (cid:26)(cid:16)
[Equation]
[Equation]
, can find an ϵ -stationary pointin O
[Equation]
Remark:
This complexity is strictly better than previous SOTA method SOX, which enjoys aniteration complexity of O (cid:16) [Equation] The sample complexity can be ob-
[Equation]
tained by multiplying the iteration complexity with B 1 B 2 .
We can see that larger B 1 or B 2 yields asmaller iteration complexity, which means that from the computational perspective if samples can beprocessed in parallel (e.g., in GPU), there is benefit of using large B 1 and/or B 2 .
However, from thesample complexity perspective, using B 1 = B 2 = 1 is the best.
The same discussion holds for othertheorems below.
However, the complexity of MSVR-v1 is still on the order of O ( ϵ − 4 ) .
Due to the biased natureof the estimated gradient, using the moving average update is not enough for achieving the SOTAcomplexity.
So, we use the technique of STORM [Cutkosky and Orabona, 2019] to update z t :
[Equation]
(8) where the projection operation is needed if using the MSVR estimator.
Now, we prove this newmethod (i.e., MSVR-v2) can obtain the optimal complexity of O ( ϵ − 3 ) .
Theorem 2 Our MSVR-v2 algorithm with α t +
= O ( mη 2 t B 1 ) ,
β t +1 =
O (cid:16)
m 2 η 2 t B 21 (cid:17) , a = O ( mB 2 B 1 ) andη t = O (cid:16) (
[Equation] [Equation] ) 2 / 3 ( a + t ) − 1 / 3 (cid:17) , can find an ϵ -stationary point in O (cid:16) mϵ − 3 B 1 √ B 2 (cid:17) iterations.
Remark: When m = 1 and f is the identity function, problem (1) reduces to the standard stochasticnon-convex optimization, whose lower bound is Ω (cid:0) ϵ − 3 (cid:1)
[Arjevani et al., 2019], indicating our
MSVR-v2 is optimal.
Next, we show that the complexity can be further improved when the objective function is convex orstrongly convex.
We note that Polyak-Łojasiewicz (PL) [Karimi et al., 2016] objectives are moregeneral than strongly convex functions, since µ -strong convexity implies the µ -PL condition.
So, wewill consider the PL condition and introduce its definition below.
Definition 2 F ( w ) satisfies the µ -PL condition if there exists µ > 0 such that:
[Equation]
Then, we derive improved rates for convex or PL objectives by using the stage-wise design given in
Algorithm 3 in the supplement.
Theorem
If the objective function satisfies the convexity or µ -PL condition, MSVR-v1 derives asample complexity of O (max( B 1 , B 2 ) ϵ − 3 ) or O (max( B 1 , B 2 ) µ − 2 ϵ − 1 ) , separately.
For MSVR-v2,the complexity can be further improved to O (cid:0) m √ B 2 ϵ − 2 (cid:1) or O (cid:0) m √ B 2 µ − 1 ϵ − 1 (cid:1) .
Remark: The complexities for MSVR-v2 are optimal, since they match the Ω (cid:0) ϵ − 2 (cid:1) and Ω (cid:0) µ − 1 ϵ − 1 (cid:1)lower bound for stochastic convex and strongly convex optimization
[Agarwal et al., 2012].
An Improved Rate for the Finite-sum Case
In this section, we consider the case that inner function
g
i is in the form of the finite-sum, i.e.,g i ( w ) =
1 n (cid:80) nj
=1 g i ( w ; ξ ij ) , so that we can compute the exact value of g i ( w ) in some iterations.
Wefirst modify our MSVR estimator to utilize the finite-sum structure.
Inspired by SVRG [Johnson and
Zhang, 2013, Zhang et al., 2013], we compute a full version of the inner function value for every Iiterations at w τ , i.e., g i ( w τ ) = 1 n (cid:80) nj =
1 g i ( w τ ; ξ ij ) for i = 1 , · · · , m , where τ mod I = 0 .
Then,in each step, we use
[Equation]
to replace g i ( w t ; ξ it ) in the origin estimator.
In this way, our MSVR estimator is changed to:
[Equation]
For this estimator, we have the following guarantee.
Compute function 10: w t +1 = w t − η z t 13: Return w τ
Lemma 3 If β ≤ 12 and βI ≤ mB 1 , by setting γ = m − B 1 B 1 (1 − β ) + (1 − β ) , we have: [Equation]
E (cid:104)
[Equation]
Remark:
Compared with Lemma 1, we remove the 2 B 1 β 2 σ 2 Bterm, which is the key to reduce thecomplexity since we can now use a larger parameter β .
To attain the optimal complexity, we modify the gradient estimator z t in a similar way:
[Equation]
where h t involves both the full gradient and the stochastic gradient (we also need to save each∇ f i ( u τ − 1 ) and calculate the full version of 1 m (cid:80) mi =1 ∇ f i ( u iτ − 1 ) ∇ g
i ( w τ ) at those steps τ ) :
[Equation]
The whole method is summarized in Algorithm 2 (named as MSVR-v3).
Next, we show that
MSVR-v3 is equipped with an optimal complexity of O ( √ nϵ − 2 ) .
Theorem 4 Our MSVR-v3 with I = mnB 1 B 2 , α = O (cid:0) B 1 B 2 mn (cid:1) , β
= O (cid:0) B 2 n (cid:1) and η = O (cid:16) B 1 √ B 2 m √ n (cid:17) ,can obtain an ϵ -stationary point in T
= O (cid:16) m √ nϵ − 2 B 1 √ B 2 (cid:17) iterations.
Remark: When m = 1 and f is the identity function, problem (2) reduces to the stochastic finite-sumoptimization, whose optimal complexity is O (cid:0) √ nϵ − 2 (cid:1)
[Fang et al., 2018, Li et al., 2021], indicatingour complexity is optimal in terms of ϵ and n .
Similarly, a better complexity can be obtained under the convexity or PL condition.
Theorem
If the objective function satisfies the convexity or µ -PL condition, the sample complexitycan be improved to O (cid:16) m
[Equation]
ϵ (cid:17) or O (cid:16) m
[Table]
nµ
[Table] [Table]
(cid:17) , respectively. Remark: It is notable that we achieve a linear convergence rate O (cid:0) log 1 ϵ (cid:1) under the PL condition,matching the current result in the single-level finite-sum problem [Li et al., 2021]
Experiments
In this section, we conduct experiments on the multi-task deep AUC maximization to evaluate theproposed methods.
For binary classification (label y = 1 or y = − 1 ), AUC maximization can be
[Table]
[Figure]
formulated as minimizing the following composite loss [Zhu et al., 2022]:
[Equation]
where a ( w ) =
E [ h w ( x ) | y = 1] , b ( w ) =
E [ h w ( x ) | y = − 1] and ℓ ( · ) is a surrogate function.
Theabove objective recovers the pairwise square loss and the min-max margin loss proposed by Yuan et al.[2020] for deep AUC maximization by setting ℓ ( · ) as the square function or squared hinge function,respectively.
When applied to multi-task classification (e.g., multiple classes), we can optimize theaveraged AUC losses over all tasks, i.e., AUC = 1 m (cid:80) mi =1 AUC ( i ) .
The nested structure only comesfrom the term ℓ ( a ( w ) − b ( w )) , and we can rewrite it as the form of FCCO problem, where
[Equation]
where D i + / − denots the positive/negative datasets of the i -th task.
Configurations.
In the experiment, we follow the setup in Zhu et al.
[2022] and set the surrogate function ℓ as squared hinge ℓ ( x ) =
12 (max { c + x, 0 } ) 2 .
We use ResNet18 as backbone network, and trainon six datasets: STL10 [Coates et al., 2011], CIFAR10 [Krizhevsky, 2009], CIFAR100 [Krizhevsky,2009], MNIST [LeCun et al., 1998], Fashion-MNIST [Xiao et al., 2017], and SVHN [Netzer et al.,2011].
We compare our methods with previous SOTA algorithm SOX
[Wang and Yang, 2022].
Forour methods, parameters α and β are searched from { 0 . 1 , 0 . 5 , 0 . 9 , 1 . 0 } .
For SOX algorithm, itsparameters β and γ are searched from the same set.
B 1 is set as 50 for CIFAR100 and 5 for otherdatasets.
Inner batch size B 2 is chosen as 128 for all methods.
We tune the learning rate from the set{ 1 e − 4 , 1 e − 3 , 2 e − 3 , 5 e − 3 , 1 e − 2 } and pick the best one for each method.
The experiments are 278 conducted on single NVIDIA Tesla M40 GPU.
Results.
Figure 1 shows the loss against the number of samples drawn by different methods, and allcurves are averaged over 5 runs.
We observe that MSVR-V1 is better than SOX on the CIFAR100dataset, and close to it on other datasets.
MSVR-v2 converges faster than SOX and MSVR-v1, andthe loss of MSVR-v3 decreases most rapidly, demonstrating a low sample complexity.
Conclusion and Future Work
In this paper, we develop a novel MSVR estimator for tracking multiple functional mappings byprobing only O (1) blocks.
Equipped with this estimator, we design three algorithms for FCCOproblems and obtain improved complexities across a spectrum of settings.
Experimental results onmulti-task deep AUC maximization also verify the effectiveness of our methods.
In future work, wewill investigate other applications that can be solved by using the proposed estimator.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[No]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This work is mainly theoretical.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] The complete proofsare provided in the supplementary.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 5.(c)
Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We use the public benchmark datasets.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable [Table] information or offensive content?
[N/A] We use the public benchmark datasets. [Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount [Table] spent on participant compensation?
[N/A]
[Table]