Asymptotically Unbiased Instance-wise Regularized Partial AUC Optimization: Theory and Algorithm
Abstract
The Partial Area Under the ROC Curve (PAUC), typically including One-way
Partial AUC (OPAUC) and Two-way Partial AUC (TPAUC), measures the averageperformance of a binary classiﬁer within a speciﬁc false positive rate and/or truepositive rate interval, which is a widely adopted measure when decision constraintsmust be considered.
Consequently, PAUC optimization has naturally attractedincreasing attention in the machine learning community within the last few years.
Nonetheless, most of the existing methods could only optimize PAUC approximately, leading to inevitable biases that are not controllable.
Fortunately, a recentwork presents an unbiased formulation of the PAUC optimization problem viadistributional robust optimization.
However, it is based on the pair-wise formulation of AUC, which suffers from the limited scalability w.r.t.
sample size and aslow convergence rate, especially for TPAUC.
To address this issue, we present asimpler reformulation of the problem in an asymptotically unbiased and instancewise manner.
For both OPAUC and TPAUC, we come to a nonconvex stronglyconcave min-max regularized problem of instance-wise functions.
On top of this,we employ an efﬁcient solver which not only enjoys a linear per-iteration computational complexity w.r.t.
the sample size but also improves the convergence rate for
TPAUC optimization from O ( T − 1 / 6 ) to O
( T − 1 / 3 ) ( T is the number of iteration).
Furthermore, we ﬁnd that the min-max reformulation also facilitates the theoreticalanalysis of generalization error as a byproduct.
Compared with the existing results,we present new error bounds that are much easier to prove and could deal withhypotheses with real-valued outputs.
Finally, extensive experiments on severalbenchmark datasets demonstrate the effectiveness of our method.
Introduction
AUC refers to the Area Under the Receiver Operating Characteristic (ROC) curve [1], where the ROCcurve is obtained by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR) of agiven classiﬁer for all possible thresholds.
Since it is insensitive to the class distribution, AUC hasbecome one of the standard metrics for long-tail, and imbalanced datasets [1, 22, 38].
Consequently,
AUC optimization has attracted increasing attention in the machine learning community ever sincethe early 2000s [13, 5, 35, 17].
Over the last two decades, research on AUC optimization hasevolved from the simplest linear models and decision trees [27, 10, 29, 41] to state-of-the-art deeplearning architectures [21, 14, 37, 43, 42, 33].
With such remarkable success, one can now easilyapply AUC optimization to deal with various real-world problems ranging from ﬁnancial frauddetection[16, 4, 23], spam email detection [24], to medical diagnosis [24, 38, 37, 43], etc.
However, in such long-tailed applications, we are often interested in a speciﬁc region in the ROC curvewhere its area is called Partial AUC (PAUC).
As illustrated in Fig.1, there are two types of PAUC.
[Figure]
FPR
TPR
TPR       0 FPR
TPR
[Figure]
[Table]
+ + n B − ) Here, One-way Partial AUC (OPAUC) measures the area within an FPR interval ( 0 ≤ FPR ≤ β );while Two-way Partial AUC (TPAUC) measures the area with FPR ≤ β , TPR ≥ α .
Unlikethe full AUC, optimizing PAUC requires selecting top-ranked or/and bottom-ranked instances,leading to a hard combinatorial optimization problem.
Many efforts have been made to solve theproblem [6, 24, 25, 20, 39].
However, the majority of them rely on full-batch optimization andthe approximation of the top (bottom)- k ranking process, which suffers from immeasurable biasesand undesirable efﬁciency.
Most recently, researchers have started to explore mini-batch PAUCoptimization for deep learning models.
[39] proposed a novel end-to-end optimization frameworkfor PAUC.
This formulation has a fast convergence rate with the help of a stochastic optimizationalgorithm, but the estimation of PAUC is still biased.
Later, [44] proposed a Distributional Robust
Optimization (DRO) framework for PAUC optimization, where the bias can be eliminated by a cleverreformulation and the compositional SGD algorithms [28].
However, they adopt the pair-wise lossfunction which has limited scalability w.r.t.
sample size and O ( (cid:15) − 4 ) /
O ( (cid:15) − 6 ) time complexity toreach the (cid:15) -stationary point for OPAUC / TPAUC .
Considering the efﬁciency bottleneck comes fromthe pair-wise formulation, we will explore the following question in this paper:
Can we design a simpler, asymptotically unbiased and instance-wise formulation to optimize OPAUC and TPAUC in an efﬁcient way?
To answer this question, we propose an efﬁcient and nearly unbiased optimization algorithm (the biasvanishes asymptotically when κ → 0 ) for regularized PAUC maximization with a faster convergenceguarantee.
The comparison with previous results are listed in Tab.1.
We consider both OPAUC and TPAUC maximization, where for OPAUC , we focus on maximizing PAUC in the region: FPR ≤ βand for TPAUC we focus on the region: FPR ≤ β and TPR ≥ α .
We summarize our contributionsbelow.
• With a proper regularization, we propose a nonconvex strongly concave min-max instancewise formulation for OPAUC and TPAUC maximization.
On top of our proposed formula-
tion, we employ an efﬁcient stochastic min-max algorithm that ﬁnds a (cid:15) -stationary point
within O ( (cid:15) − 3 ) iterations.
• We conduct a generalization analysis of our proposed methods.
Our instance-wise reformulation can overcome the interdependent issue of the original pair-wise generalization analysis.
The proof is much easier than the existing results.
Moreover, compared with [25, 39], it canbe applied to any real-valued hypothesis functions other than the hard-threshold functions.• We conduct extensive experiments on multiple imbalanced image classiﬁcation tasks.
Theexperimental results speak to the effectiveness of our proposed methods.
Preliminaries
Notations.
In this section, we give some deﬁnitions and preliminaries about OPAUC and TPAUC .
Let X ⊆
R
d be the input space, Y = { 0 , 1 } be the label space.
We denote D P and D N as positiveand negative instance distribution, respectively.
Let S =
{ z = ( x i , y i ) } ni =1 be a set of trainingdata drawn from distribution D Z , where n is the number of samples.
Let P ( N resp.)
be a setof positive (negative resp.)
instances in the dataset.
In this paper we only focus on the scoringfunctions f : X (cid:55)→ [0 , 1] .
The output range can be simply implemented by any deep neural networkwith sigmoid outputs.
Standard AUC.
The standard AUC calculates the entire area under the ROC curve.
For mathematicalconvenience, our arguments start with the pair-wise reformulation of AUC.
Speciﬁcally, as shownin [1], AUC measures the probability of a positive instance having a higher score than a negativeinstance:
[Equation]
OPAUC .
As mentioned in the introduction, instead of considering the entire region of ROC, wefocus on two forms of PAUC, namely TPAUC and OPAUC.
According to [6], OPAUC is equivalentto the probability of a positive instance x being scored higher than a negative instance x (cid:48) within thespeciﬁc
[Equation]
Practically, we do not know the exact data distributions D P , D N to calculate (2).
Therefore, weturn to the empirical estimations of (2).
Given a ﬁnite dataset S with n instances, let n + , n − be thenumbers of positive/negative instances, respectively.
For the OPAUC , its empirical estimation couldbe expressed as [24]:
[Equation]
where n β − = (cid:98) n − · β (cid:99) ; x (cid:48) [ j ] denotes the j -th largest score among negative samples; (cid:96) 0 , 1 ( t ) is the 0 − 1loss, which returns 1 if t < 0 and 0 otherwise.
TPAUC .
More recently, [36] argued that an efﬁcient classiﬁer should have low FPR and high
TPR simultaneously.
Therefore, we also study a more general variant called Two-way Partial AUC( TPAUC ), where the restricted regions satisfy TPR ≥ α and FPR ≤ β .
Similar to OPAUC ,
TPAUC measures the probability that a positive instance x ranks higher than a negative instance x (cid:48)where f ( x ) ∈ [0 , η α ( f )] , s .
t .
Pr x ∼D P [ f ( x ) ≤ η α ]
= α, f ( x (cid:48) ) ∈
[ η β ( f ) , [Equation]
[Equation] N
[Equation] β .
[Equation]
Similarly to OPAUC , for the TPAUC , we adopt its empirical estimation [36, 39]:
[Equation]
where n α + = (cid:98) n + · α (cid:99) , and x [ i ] is i -th smallest score among all positive instances.
Problem Formulation
In this section, we introduce how to optimize OPAUC and TPAUC in an asymptotically unbiasedinstance-wise manner.
Note that (3) and (5) are hard to optimize since it is complicated to determinethe positive quantile η α ( f ) and the negative quantile η β ( f ) .
So we can not obtain the bottom-rankedpositive instances and top-ranked negative instances directly.
In this section, we will elaborate onhow to tackle these challenges.
3.1 Optimizing the OPAUC
According to Eq.(3), given a surrogate loss (cid:96) and the ﬁnite dataset S , maximizing OPAUC andˆAUC β ( f, S ) is equivalent to solving the following problems, respectively:
[Equation]
Step 1: Instance-wise Reformulation.
Here, to simplify the reformulation, we will use the mostpopular surrogate squared loss (cid:96) ( x ) =
(1 − x ) 2 .
Under this setting, the following theorem showsan instance-wise reformulation of the OPAUC optimization problem (please see Appendix.
G for theproof):
Theorem 1.
Assuming that f ( x ) ∈ [0 , 1] , ∀ x ∈ X , F op ( f, a, b, γ, t, z ) is deﬁned as:
[Equation]
where y = 1 for positive instances, y = 0 for negative instances and we have the followingconclusions:(a)
( Population Version .)
We have:
[Equation]
where η β ( f ) = arg min η β ∈ R [ E x (cid:48) ∼D N [ I [ f ( x (cid:48) )
≥ η β ]] =
β ] .(b) ( Empirical Version .)
Moreover, given a training dataset S with sample size n , denote:
[Equation]
where ˆ η β ( f ) is the empirical quantile of the negative instances in S .
We have:
[Equation]
Step 2: Differentiable Sample Selection.
Thm.1 provides a support to convert the pair-wise lossinto instance-wise loss for OPAUC .
However, the min-max problem Eq.(10) is still difﬁcult to solvedue to the operation I f ( x (cid:48) ) ≥ η β ( f ) , which requires selecting top-ranked negative instances.
To make 123 the sample selection process differentiable, we adopt the following lemma.
Lemma 1.
(cid:80) k
i =1 x [ i ] is a convex function of ( x 1 , · · · , x n ) where x [ i ] is the top-i element of a set { x 1 , x 2 , · · · , x n } .
Furthermore, for x i , i = 1 , · · · , n , we have (cid:80) ki
=1 x [ i ] =
min s { ks + (cid:80) ni
=1 [ x i − s ] + } , where [ a ] + = max { 0 , a } .
The population version is E x [ x · I [ x ≥ η ( α )] ] = min s E x [ αs + [ x − s ] + ] where η ( α ) = arg min η ∈ R E
x [ I [ x ≥ η ] = α ] (Please see Appendix. G for the proof). Lem.1 proposes an Average Top- k (ATk) loss which is the surrogate loss for top- k loss to eliminatethe sorting problem.
Optimizing the ATk loss is equivalent to selecting top-ranked instances.
Actually,for R β ( f ) , we can just reformulate it as an Average Top- k (ATk) loss.
Denote (cid:96) − ( x (cid:48) ) =
( f ( x (cid:48) ) −b ) {+
+} + 2(1 + γ ) f
( x (cid:48) ) .
In the proof of the next theorem, we will show that (cid:96) − ( x (cid:48) ) is an increasingfunction w.r.t.
f ( x (cid:48) ) , namely:
[Equation]
The similar result holds for ˆ R β ( f, S ) .
Then, we can reach to Thm.2 (please see Appendix.
G for theproof):
Theorem 2.
Assuming that f ( x ) ∈ [0 , 1] , for all x ∈ X , we have the equivalent optimization for
OPAUC :
[Equation]
[Equation]
[Equation]
where Ω γ =
[ b − 1 , 1] , Ω s (cid:48) =
[0 , 5] and
G op
[Equation]
f
[Equation]
a, b
[Equation]
γ
[Equation]
z
[Equation]
Step 3: Asymptotically Unbiased Smoothing.
Even with Thm.2, it is hard to optimize the minmax-min formulation in Eq.(13).
A solution is to swap the order max γ and min s (cid:48) to reformulate it asa min-max problem.
The key obstacle to this idea is the non-smooth function [ · ] + .
To avoid the [ · ] + ,we apply the softplus function [12]:
[Equation]
as a smooth surrogate.
It is easy to show that r κ ( x ) κ →∞ → [ x ] + .
Denote G κop ( f, a, b, γ, z , s (cid:48) ) thesurrogate objective where the [ · ] + in G op ( f, a, b, γ, z , s (cid:48) ) is replaced with r κ ( · ) .
We then proceedto solve the surrogate problem:
[Equation]
respectively for the population and empirical version.
In Appendix.
C, we will proof that such aapproximation has a convergence rate O (1 /κ ) .
Step 4: The Regularized Problem.
It is easy to check that r κ ( x ) has a bounded second-order derivation.
In this way, we can regard G κop ( f, a, b, γ, z , s (cid:48) ) as a weakly-concave function [3] of γ .
Byemploying an (cid:96) 2 regularization, we turn to a regularized form:
[Equation]
With a sufﬁciently large ω , G κ,ωop ( f, a, b, γ, z , s (cid:48) ) is strongly-concave w.r.t.
γ when all the othervariables are ﬁxed.
Note that the regularization scheme will inevitably bias.
As a very general result,regularization will inevitably induce bias.
However, it is known to be a necessary building blockto stabilize the solutions and improve generalization performance.
We then reach a min-maxproblem in the ﬁnal step.
Step 5: Min-Max Swapping.
According to min-max theorem [3], if we replace G κop ( f, a, b, γ, z , s (cid:48) )with G κ,ωop ( f, a, b, γ, z , s (cid:48) ) , the surrogate optimization problem satisﬁes:
[Equation]
where G κ,ωop = G κ,ωop ( f, a, b, γ, z , s (cid:48) )
In this sense, we come to a regularized non-convex stronglyconcave problem.
In Sec.4, we will employ an efﬁcient solver to optimize the parameters.
3.2 Optimizing the TPAUC
According to Eq.(5), given a surrogate loss (cid:96) and ﬁnite dataset S , maximizing TPAUC andˆAUC α,β ( f, S ) is equivalent to solving the following problems, respectively:
[Equation] [Equation]
Due to the limited space, we present the result directly, please refer to Appendix.
E for more details.
Similar to OPAUC , we apply the function r κ ( x ) , regularization ωγ 2 and min-max theorem to solvethe problem.
In this sense, we can use
[Equation]
where Ω γ = [max {− a, b − 1 , 1 } ] and
[Equation]
to minimize R β,α ( f ) , and ˆ R β,α ( f ) , respectively.
Here:
G
[Equation]
,ωtp
[Equation]
f, a, b, γ, z , s, s
[Equation]
αs
[Equation]
According to Thm.2 of [32], we have the following corollary:
Corollary 1.
We can reformulate Eq.
(18) and Eq.
(23) as an off-the-shelf mini-max problem where the coupled constraint is replaced with the Lagrange multipliers ( θ b for OPAUC, θ b , θ a for
TPAUC).
For OPAUC :
[Equation]
For TPAUC :
[Equation]
The tight constraint θ b ∈ [0 , M 1 ] , θ b ∈ [0 , M 2 ] , θ a ∈ [0 , M 3 ] comes from the fact that optimum θ b , θ a are both ﬁnite since the objective function is bounded from above.
In the experiments, to make sure that M 1 , M 2 , M 3 are large enough, we set them as M 1 =
M 2 = M 3 = 10 9 .
Training Algorithm
According to the derivations in the previous sections, our goal is then to solve the resulting empiricalmin-max optimization problems in Eq.(18) and Eq.(22).
It is easy to check that they are stronglyconcave w.r.t γ whenever κ ≤ 2 + 2 ω , when ( f ( x ) , a, b ) ∈ [0 , 1] 3 , γ ∈ = [ − 1 , 1] , s ∈ Ω s , s (cid:48) ∈
Ω s (cid:48) .
Therefore, we can adopt the nonconvex strongly concave min-max optimization algorithms to solvethese problems [15].
In this section, following the work [15], we employ an accelerated stochasticgradient descent ascent (ASGDA) method to solve the min-max optimization problem.
We denoteθ ∈ R d as the parameters of function f , τ = { θ , a, b, s, s (cid:48) , θ p , θ n } ∈ Ω τ as the variables for the outermin-problem.
Algorithm 1 shows the framework of our algorithm ( Due the the page limitation, weput the algorithm and convergence rate theorem in Appendix.B ).
There are two key steps.
At Line 5-6 of Algorithm 1, variables τ t +1 and γ t +1 are updated in a momentum way.
Moreover, theconvex combination ensures that they are always feasible given that the initial solution is feasible.
At
Line 9-10 , using the momentum-based variance reduced technique, we can estimate the stochasticﬁrst-order partial gradients v t and w t in a more stable manner.
Generalization Analysis
In this section, we theoretically analyze the generalization performance of our proposed estimatorsfor OPAUC (please see the Appendix.
H for the TPAUC ).
According to Thm.2 in Sec.3, we knowthat the generalization error of OPAUC with a surrogate loss (cid:96) can be measured as:
[Equation]
[Equation]
and
[Equation]
Following the ERM paradigm, to prove the uniform convergence result over a hypothesis class F ofthe scoring function f , we need to show that:
[Equation]
According to the aforementioned discussion, we only need to prove that:
[Equation]
To prove this, we need to deﬁne the measure of the complexity of the class F .
Here we adopt the
Radermacher complexity (cid:60) as in [2].
Speciﬁcally, we come to the following deﬁnition:
Deﬁnition 1.
The empirical Rademacher complexity of positive and negative instances with respectto S is deﬁned as:
[Equation]
29) where ( σ 1 , · · · , σ m ) are independent uniform random variables taking values in {− 1 , +1 } .
Finally, we come to the generalization bound as follows:
Theorem 3.
For any δ > 0 , with probability at least 1 − δ over the draw of an i.i.d.
sample set S ofsize m , for all f ∈ F we have:
[Equation]
Remark 1.
Although the results we obtain are similar to some previous studies.
[39, 25], ourgeneralization analysis is simpler and does not require complex error decomposition.
Moreover,our results hold for all real-valued hypothesis class with outputs in [0 , 1] , while the previous results[39, 25] only hold for hard-threshold functions.
Experiment
In this section, we conduct a series of experiments on different datasets for both OPAUC and
TPAUC optimization.
Due to space limitations, please refer to the Appendix.
F for the details ofimplementation and competitors.
6.1 Setups
We adopt three imbalanced binary classiﬁcation datasets: CIFAR-10-LT [8], CIFAR-100-LT [19] and
Tiny-ImgaeNet-200-LT following the instructions in [39], where the binary datasets are constructedby selecting one super category as positive class and the other categories as negative class.
Please see
Appendix.F for more details.
The evaluation metrics in experiments are ˆAUC β and ˆAUC α,β .
6.2 Overall Results
In Tab.2, Tab.3 and Tab.4, we record the performance on test sets of all the methods on three subsetsof CIFAR-10-LT, CIFAR-100-LT, and Tiny-Imagent-200-LT.
Each method is tuned independently for
OPAUC and TPAUC metrics.
From the results, we make the following remarks: (1) Our proposedmethods outperform all the competitors on all the datasets in most cases, except for the CIFAR-100LT Subset 3.
Even for failure cases, our methods attain fairly competitive results compared with thecompetitors for OPAUC and TPAUC .
In addition, we can see that the normal AUC optimizationmethod AUC-M has poor performance under PAUC metric.
This demonstrates the necessity ofdeveloping the PAUC optimization algorithm.
(3) Approximation methods like DROS, AUC-poly,and AUC-exp have lower performance than the unbiased algorithms.
Above all, the empirical resultssuggest the effectiveness of our proposed method.
The results in Tab.4 show that our algorithmcan reach comparable performance with AGD-SBCD in this case.
That means our algorithm can beapplied to OPAUC optimization within a ﬁxed FPR range.
[Table]
6.3 Convergence Analysis
In the convergence experiments, for sake of fairness, we did not use warm-up.
All algorithms usehyperparameters in the performance experiments.
[-We show the plots of training convergence in Fig.
-]
We show the plots of training convergence in Fig.4and Fig.5 on CIFAR-10 for both OPAUC and TPAUC .
Due to the space limitation, the other resultscould be found in Appendix.
F.
According to the ﬁgures, we can make the following observations: (1)
Our algorithm and DROE converge faster than other methods for OPAUC .
However, for TPAUCoptimization, the DROE converges very slowly due to its complicated algorithm, while our method [Table] [Table] [Table]
still shows the best convergence property in most cases.
(2) It’s notable that our algorithm convergesto stabilize after twenty epochs in most cases.
That means our method has better stability in practice.      HSRFKV         2 3$ 8&   W U D L Q L QJ  V H W  3$8&,$8&SRO\$8&H[S623$623$60%&($8&0$*'6%&' (a) CIFAR-100-Long-Tail-       HSRFKV         2 3$ 8&   W U D L Q L QJ  V H W  3$8&,$8&SRO\$8&H[S623$623$60%&($8&0$*'6%&' (b) CIFAR-100-Long-Tail-
      HSRFKV          2 3$ 8&   W U D L Q L QJ  V H W  3$8&,$8&SRO\$8&H[S623$623$60%&($8&0$*'6%&' ( [Figure]  Conclusion
In this paper, we focus on designing an efﬁcient and asymptotically unbiased algorithm for PAUC.
We propose a nonconvex strongly concave min-max instance-wise formulation for OPAUC and
TPAUC .
In this way, we incorporate the instances selection into the loss calculation to eliminatethe score ranking challenge.
For OPAUC and TPAUC , we employ an efﬁcient stochastic min-maxalgorithm that ensures we can ﬁnd a (cid:15) -ﬁrst order saddle point after O ( (cid:15) − 3 ) iterations.
Moreover,we present a theoretical analysis of the generalization error of our formulation.
Our conclusionmay contribute to future work about AUC generalization.
Finally, empirical studies over a range oflong-tailed benchmark datasets speak to the effectiveness of our proposed algorithm.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[No] We have
not discovered any negative societal impacts in our study.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No] Our experiments are very time consuming.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[No] Because we use the open source software and data.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table] If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable? [N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A]
[Table]