Asymptotically Unbiased Instance-wise Regularized Partial AUC Optimization: Theory and Algorithm Anonymous Author(s)
AffiliationAddress
Abstract
The Partial Area Under the ROC Curve (PAUC), typically including One-way
Partial AUC (OPAUC) and Two-way Partial AUC (TPAUC), measures the averageperformance of a binary classifier within a specific false positive rate and/or truepositive rate interval, which is a widely adopted measure when decision constraintsmust be considered.
Consequently, PAUC optimization has naturally attractedincreasing attention in the machine learning community within the last few years.
Nonetheless, most of the existing methods could only optimize PAUC approximately, leading to inevitable biases that are not controllable.
Fortunately, a recentwork presents an unbiased formulation of the PAUC optimization problem via distributional robust optimization.
However, it is based on the pair-wise formulationof AUC, which suffers from the limited scalability w.r.t.
sample size and a slowconvergence rate, especially for TPAUC.
To address this issue, we present a simplerreformulation of the problem in an asymptotically unbiased and instance-wisemanner.
For both OPAUC and TPAUC, we come to a nonconvex strongly concavemin-max regularized problem of instance-wise functions.
On top of this, we employ an efficient solver enjoys a linear per-iteration computational complexityw.r.t.
the sample size and a time-complexity of O ( ϵ − 1 / 3 ) to reach a ϵ stationarypoint .
Furthermore, we find that the min-max reformulation also facilitates thetheoretical analysis of generalization error as a byproduct.
Compared with theexisting results, we present new error bounds that are much easier to prove andcould deal with hypotheses with real-valued outputs.
Finally, extensive experimentson several benchmark datasets demonstrate the effectiveness of our method.
Introduction
AUC refers to the Area Under the Receiver Operating Characteristic (ROC) curve [1], where the ROCcurve is obtained by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR) of agiven classifier for all possible thresholds.
Since it is insensitive to the class distribution, AUC hasbecome one of the standard metrics for long-tail, and imbalanced datasets [1, 22, 38].
Consequently,
AUC optimization has attracted increasing attention in the machine learning community ever sincethe early 2000s [13, 5, 35, 17].
Over the last two decades, research on AUC optimization hasevolved from the simplest linear models and decision trees [27, 10, 29, 41] to state-of-the-art deeplearning architectures [21, 14, 37, 43, 42, 33].
With such remarkable success, one can now easilyapply AUC optimization to deal with various real-world problems ranging from financial frauddetection[16, 4, 23], spam email detection [24], to medical diagnosis [24, 38, 37, 43], etc.
However, in such long-tailed applications, we are often interested in a specific region in the ROC curvewhere its area is called Partial AUC (PAUC).
As illustrated in Fig.1, there are two types of PAUC.
Here, One-way Partial AUC (OPAUC) measures the area within an FPR interval ( 0 ≤ FPR ≤ β );
[Figure]
FPR
TPR
TPR       0 FPR
TPR
[Figure]
SOPA
[Table]
[Table]
while Two-way Partial AUC (TPAUC) measures the area with FPR ≤ β [Table] TPR ≥ α .
Unlikethe full AUC, optimizing PAUC requires selecting top-ranked or/and bottom-ranked instances,leading to a hard combinatorial optimization problem.
Many efforts have been made to solve theproblem [6, 24, 25, 20, 39].
However, the majority of them rely on full-batch optimization andthe approximation of the top (bottom)- k ranking process, which suffers from immeasurable biasesand undesirable efficiency.
Most recently, researchers have started to explore mini-batch PAUCoptimization for deep learning models.
[39] proposed a novel end-to-end optimization frameworkfor PAUC.
This formulation has a fast convergence rate with the help of a stochastic optimizationalgorithm, but the estimation of PAUC is still biased.
Later, [44] proposed a Distributional Robust
Optimization (DRO) framework for PAUC optimization, where the bias can be eliminated by a cleverreformulation and the compositional SGD algorithms [28].
However, they adopt the pair-wise lossfunction which has limited scalability w.r.t.
sample size and O ( ϵ − 4 ) /
O ( ϵ − 6 ) time complexity toreach the ϵ -stationary point for OPAUC / TPAUC .
Considering the efficiency bottleneck comes fromthe pair-wise formulation, we will explore the following question in this paper:
Can we design a simpler, nearly asymptotically unbiased and instance-wise formulation to optimize OPAUC and TPAUC in an efficient way?
To answer this question, we propose an efficient and nearly unbiased optimization algorithm (the biasvanishes asymptotically when κ → 0 ) for regularized PAUC maximization with a faster convergenceguarantee.
The comparison with previous results are listed in Tab.1.
We consider both OPAUC and
TPAUC maximization, where for OPAUC , we focus on maximizing PAUC in the region: FPR ≤ βand for TPAUC we focus on the region: FPR ≤ β and TPR ≥ α .
We summarize our contributionsbelow.
• With a proper regularization, we propose a nonconvex strongly concave min-max instancewise formulation for OPAUC and TPAUC maximization.
On top of our proposed formula-
tion, we employ an efficient stochastic min-max algorithm that finds a ϵ -stationary pointwithin O ( ϵ − 3 ) iterations.
• We conduct a generalization analysis of our proposed methods.
Our instance-wise reformulation
can overcome the interdependent issue of the original pair-wise generalization analysis.
The proof is much easier than the existing results.
Moreover, compared with [25, 39], it canbe applied to any real-valued hypothesis functions other than the hard-threshold functions.• We conduct extensive experiments on multiple imbalanced image classification tasks.
Theexperimental results speak to the effectiveness of our proposed methods.
Preliminaries
Notations.
In this section, we give some definitions and preliminaries about OPAUC and TPAUC .
Let X ⊆
R
d be the input space, Y = { 0 , 1 } be the label space.
We denote D P and D N as positiveand negative instance distribution, respectively.
Let S =
{ z = ( x i , y i ) } ni =1 be a set of trainingdata drawn from distribution D Z , where n is the number of samples.
Let P ( N resp.)
be a setof positive (negative resp.)
instances in the dataset.
In this paper we only focus on the scoringfunctions f : X (cid:55)→ [0 , 1] .
The output range can be simply implemented by any deep neural networkwith sigmoid outputs.
Standard AUC.
The standard AUC calculates the entire area under the ROC curve.
For mathematicalconvenience, our arguments start with the pair-wise reformulation of AUC.
Specifically, as shownin [1], AUC measures the probability of a positive instance having a higher score than a negativeinstance:
[Equation]
OPAUC .
As mentioned in the introduction, instead of considering the entire region of ROC, wefocus on two forms of PAUC, namely TPAUC and OPAUC.
According to [6], OPAUC is equivalentto the probability of a positive instance x being scored higher than a negative instance x ′ within thespecific range f ( x ′ ) ∈ [ η β ( f ) , 1] s . t .
Pr x ′ ∼D N [ f ( x ′ ) ≥ η β ] = β :
[Equation]
Practically, we do not know the exact data distributions D P , D N to calculate (2).
Therefore, weturn to the empirical estimations of (2).
Given a finite dataset S with n instances, let n + , n − be thenumbers of positive/negative instances, respectively.
For the OPAUC , its empirical estimation couldbe expressed as [24]:
[Equation]
where n β − = ⌊ n − · β ⌋ ; x ′ [ j ] denotes the j -th largest score among negative samples; ℓ 0 , 1 ( t ) is the 0 − 1loss, which returns 1 if t < 0 and 0 otherwise.
TPAUC .
More recently, [36] argued that an efficient classifier should have low FPR and high
TPR simultaneously.
Therefore, we also study a more general variant called Two-way Partial AUC( TPAUC ), where the restricted regions satisfy TPR ≥ α and FPR ≤ β .
Similar to OPAUC ,
TPAUC measures the probability that a positive instance x ranks higher than a negative instance x ′where f ( x ) ∈ [0 , η α ( f )] , s .
t .
Pr x ∼D P [ f ( x ) ≤ η α ]
= α, f ( x ′ ) ∈
[ η β ( f ) , 1] s .
t .
Pr x ′ ∼D N
[ f ( x ′ ) ≥ η β ] =β .
[Equation]
Similarly to OPAUC , for the TPAUC , we adopt its empirical estimation [36, 39]:
[Equation]
where n α + = ⌊ n + · α ⌋ , and x [ i ] is i -th smallest score among all positive instances.
Problem Formulation
In this section, we introduce how to optimize OPAUC and TPAUC in an asymptotically unbiasedinstance-wise manner.
Note that (3) and (5) are hard to optimize since it is complicated to determine the positive quantile η α ( f ) and the negative quantile η β ( f ) .
So we can not obtain the bottom-rankedpositive instances and top-ranked negative instances directly.
In this section, we will elaborate onhow to tackle these challenges.
3.1 Optimizing the OPAUC
[Figure] According to Eq.(3), given a surrogate loss ℓ and the finite dataset S , maximizing OPAUC andˆAUC β ( f, S ) is equivalent to solving the following problems, respectively:
[Equation] [Equation]
Step 1: Instance-wise Reformulation.
Here, to simplify the reformulation, we will use the mostpopular surrogate squared loss ℓ ( x ) =
(1 − x ) 2 .
Under this setting, the following theorem showsan instance-wise reformulation of the OPAUC optimization problem (please see Appendix.
G for theproof):
Theorem 1.
Assuming that f ( x ) ∈ [0 , 1] , ∀ x ∈ X , F op ( f, a, b, γ, t, z ) is defined as:
[Equation]
where y = 1 for positive instances, y = 0 for negative instances and we have the followingconclusions:
[Figure]
(a) ( Population Version .)
We have:
[Equation]
where η β ( f ) =
arg min η β ∈ R [ E x ′
∼D N [ I [ f ( x ′ ) ≥ η β ]]
= β ] .(b)
( Empirical Version .)
Moreover, given a training dataset S with sample size n , denote:
[Equation]
where ˆ η β ( f ) is the empirical quantile of the negative instances in S .
We have:
[Equation]
Step 2: Differentiable Sample Selection.
Thm.1 provides a support to convert the pair-wise lossinto instance-wise loss for OPAUC .
However, the min-max problem Eq.(10) is still difficult to solvedue to the operation I f ( x ′ ) ≥ η β ( f ) , which requires selecting top-ranked negative instances.
To make 122 the sample selection process differentiable, we adopt the following lemma.
Lemma 1.
(cid:80) ki
=1 x [ i ] is a convex function of ( x 1 , · · · , x n ) where x [ i ] is the top-i element of a set { x 1 , x 2 , · · · , x n } .
Furthermore, for x i , i = 1 , · · · , n , we have (cid:80) ki
= 1 x [ i ] =
min s { ks + (cid:80)
{+ni =
+} [ x i − s ] + } , where [ a ] + = max { 0 , a } .
The population version is E x [ x · I [ x ≥ η ( α )] ]
=
min s E x [ αs + [ x − s ] + ] where η ( α ) =
arg min η ∈ R E
x
[ I [ x ≥ η ] = α ] (Please see Appendix.
G for the proof).
Lem.1 proposes an Average Top- k (ATk) loss which is the surrogate loss for top- k loss to eliminatethe sorting problem.
Optimizing the ATk loss is equivalent to selecting top-ranked instances.
Actually,for R β ( f ) , we can just reformulate it as an Average Top- k (ATk) loss.
Denote ℓ − ( x ′ ) =
( f ( x ′ ) −b ) [-
-]2 + 2(1 + γ ) f
( x ′ ) .
In the proof of the next theorem, we will show that ℓ − ( x ′ ) is an increasingfunction w.r.t.
f ( x ′ ) , namely:
[Equation]
The similar result holds for ˆ R β ( f, S ) .
Then, we can reach to Thm.2 (please see Appendix.
G for theproof):
Theorem 2.
Assuming that f ( x ) ∈ [0 , 1] , for all x ∈ X , we have the equivalent optimization for
OPAUC :
[Equation] [Equation]
where Ω γ =
[ b − 1 , 1] , Ω s ′ =
[0 , 5] and
[Equation]
Step 3: Asymptotically Unbiased Smoothing.
Even with Thm.2, it is hard to optimize the minmax-min formulation in Eq.(13).
A solution is to swap the order max γ and min s ′ to reformulate it asa min-max problem.
The key obstacle to this idea is the non-smooth function [ · ] + .
To avoid the [ · ] + ,we apply the softplus function [12]:
[Equation]
as a smooth surrogate.
It is easy to show that r κ ( x ) κ [Equation] → [ x ] + .
Denote G κop ( f, a, b, γ, z , s ′ ) thesurrogate objective where the [ · ] + in G op ( f, a, b, γ, z , s ′ ) is replaced with r κ ( · ) .
We then proceedto solve the surrogate problem:
[Equation]
respectively for the population and empirical version.
In Appendix.
C, we will proof that such aapproximation has a convergence rate O (1 /κ ) .
Step 4: The Regularized Problem.
It is easy to check that r κ ( x ) has a bounded second-order derivation.
In this way, we can regard G κop ( f, a, b, γ, z , s ′ ) as a weakly-concave function [3] of γ .
Byemploying an ℓ 2 regularization, we turn to a regularized form:
[Equation]
With a sufficiently large ω , G κ,ωop ( f, a, b, γ, z , s ′ ) is strongly-concave w.r.t.
γ when all the othervariables are fixed.
Note that the regularization scheme will inevitably bias.
As a very general result,regularization will inevitably induce bias.
However, it is known to be a necessary building blockto stabilize the solutions and improve generalization performance.
We then reach a min-maxproblem in the final step.
Step 5: Min-Max Swapping.
According to min-max theorem [3], if we replace G κop ( f, a, b, γ, z , s ′ )with G κ,ωop ( f, a, b, γ, z , s ′ ) , the surrogate optimization problem satisfies:
[Equation]
where G κ,ωop = G κ,ωop ( f, a, b, γ, z , s ′ )
In this sense, we come to a regularized non-convex stronglyconcave problem.
In Sec.4, we will employ an efficient solver to optimize the parameters.
3.2 Optimizing the TPAUC
According to Eq.(5), given a surrogate loss ℓ and finite dataset S , maximizing TPAUC andˆAUC α,β ( f, S ) is equivalent to solving the following problems, respectively:
[Equation] [Equation]
Due to the limited space, we present the result directly, please refer to Appendix.
E for more details.
Similar to OPAUC , we apply the function r κ ( x ) , regularization ωγ 2 and min-max theorem to solvethe problem.
In this sense, we can use
[Equation]
where Ω γ =
[max {− a, b − 1 , 1 } ] and
[Equation]
to minimize R β,α ( f ) , and ˆ R β,α ( f ) , respectively.
Here:
[Equation]
According to Thm.2 of [32], we have the following corollary:
Corollary 1.
We can reformulate Eq.
(18) and Eq.
(23) as an off-the-shelf mini-max problem where the coupled constraint is replaced with the Lagrange multipliers ( θ b for OPAUC, θ b , θ a for
TPAUC).
For OPAUC :
[Equation]
For TPAUC :
[Equation]
The tight constraint θ b ∈ [0 , M 1 ] , θ b ∈ [0 , M 2 ] , θ a ∈ [0 , M 3 ] comes from the fact that optimum θ b , θ a are both finite since the objective function is bounded from above.
In the experiments, to make sure that M 1 , M 2 , M 3 are large enough, we set them as M 1 =
M 2 = M 3 = 10 9 .
Training Algorithm
According to the derivations in the previous sections, our goal is then to solve the resulting empiricalmin-max optimization problems in Eq.(18) and Eq.(22).
It is easy to check that they are stronglyconcave w.r.t γ whenever κ ≤ 2 + 2 ω , when ( f ( x ) , a, b ) ∈ [0 , 1] 3 , γ ∈ =
[ − 1 , 1] , s ∈ Ω s , s ′ ∈ Ω s ′ .
Therefore, we can adopt the nonconvex strongly concave min-max optimization algorithms to solvethese problems [15].
In this section, following the work [15], we employ an accelerated stochasticgradient descent ascent (ASGDA) method to solve the min-max optimization problem.
We denoteθ ∈ R d as the parameters of function f , τ = { θ , a, b, s, s ′ , θ p , θ n } ∈ Ω τ as the variables for theouter min-problem.
Algorithm 1 shows the framework of our algorithm ( Due the the page limitation, we put the algorithm and convergence rate theorem in Appendix.B ).
There are two key steps.
At
Line 5-6 of Algorithm 1, variables τ t +1 and γ t +1 are updated in a momentum way.
Moreover, theconvex combination ensures that they are always feasible given that the initial solution is feasible.
At
Line 9-10 , using the momentum-based variance reduced technique, we can estimate the stochasticfirst-order partial gradients v t and w t in a more stable manner.
Generalization Analysis
In this section, we theoretically analyze the generalization performance of our proposed estimatorsfor OPAUC (please see the Appendix.
H for the TPAUC ).
According to Thm.2 in Sec.3, we knowthat the generalization error of OPAUC with a surrogate loss ℓ can be measured as:
[Equation]
and
[Equation]
Following the ERM paradigm, to prove the uniform convergence result over a hypothesis class F ofthe scoring function f , we need to show that:
[Equation]
According to the aforementioned discussion, we only need to prove that:
[Equation]
To prove this, we need to define the measure of the complexity of the class F .
Here we adopt the
Radermacher complexity ℜ as in [2].
Specifically, we come to the following definition:
Definition 1.
The empirical Rademacher complexity of positive and negative instances with respectto S is defined as:
[Equation]
where ( σ 1 , · · · , σ m ) are independent uniform random variables taking values in {− 1 , +1 } .
Finally, we come to the generalization bound as follows:
Theorem 3.
For any δ > 0 , with probability at least 1 − δ over the draw of an i.i.d.
sample set S ofsize m , for all f ∈ F we have:
[Equation]
Remark 1.
Although the results we obtain are similar to some previous studies.
[39, 25], ourgeneralization analysis is simpler and does not require complex error decomposition.
Moreover,our results hold for all real-valued hypothesis class with outputs in [0 , 1] , while the previous results[39, 25] only hold for hard-threshold functions.
Experiment
In this section, we conduct a series of experiments on different datasets for both OPAUC and
TPAUC optimization.
Due to space limitations, please refer to the Appendix.
F for the details ofimplementation and competitors.
6.1 Setups
We adopt three imbalanced binary classification datasets: CIFAR-10-LT [8], CIFAR-100-LT [19] and
Tiny-ImgaeNet-200-LT following the instructions in [39], where the binary datasets are constructedby selecting one super category as positive class and the other categories as negative class.
Please see
Appendix.F for more details.
The evaluation metrics in experiments are ˆAUC β and ˆAUC α,β .
6.2 Overall Results
In Tab.2, Tab.3, we record the performance on test sets of all the methods on three subsets of
CIFAR-10-LT, CIFAR-100-LT, and Tiny-Imagent-200-LT.
Each method is tuned independently for
OPAUC and TPAUC metrics.
From the results, we make the following remarks: (1) Our proposedmethods outperform all the competitors in most cases.
Even for failure cases, our methods attainfairly competitive results compared with the competitors for OPAUC and TPAUC .
In addition,we can see that the normal AUC optimization method AUC-M has less reasonable performance under
PAUC metric.
This demonstrates the necessity of developing the PAUC optimization algorithm.
Approximation methods SOPA-S, AUC-poly, and AUC-exp have lower performance than theunbiased algorithm SOPA and our instance-wise algorithm PAUCI in most cases.
Above all, theexperimental results show the effectiveness of our proposed method. [Table] [Table]
SOPA-S [44]
[Table]
[Table]
[Table]
PAUCI
[Table]
6.3 Convergence Analysis
In the convergence experiments, for sake of fairness, we did not use warm-up.
All algorithms usehyperparameters in the performance experiments.
We show the plots of training convergence in Fig.4and Fig.5 on CIFAR-10 for both OPAUC and TPAUC .
Due to the space limitation, the other results could be found in Appendix.
F.
According to the figures, we can make the following observations:(1)
Our algorithm and SOPA converge faster than other methods for OPAUC .
However, for
TPAUC optimization, the SOPA converges very slowly due to its complicated algorithm, whileour method still shows the best convergence property in most cases .
(2) It’s notable that ouralgorithm converges to stabilize after twenty epochs in most cases.
That means our method has betterstability in practice.
[Figure] U D
[Figure]
L
[Figure]
[Figure]
    
[Figure]
[Figure]
U D L
[Figure]
L QJ  V
[Figure]
W  3$8&,$8&SRO\$8&H[S623$623$60%&($8&
[Figure]
      HSRFKV 
    7 3$ 8&   W U D L Q L QJ  V H W  3$8&,$8&SRO\$8&H[S623$623$60%&($8& (
[Figure]
      HSRFKV      
  7 3$ 8&   W U D L Q L QJ  V H W
[Figure]
Conclusion
In this paper, we focus on designing an efficient and asymptotically unbiased algorithm for PAUC.
We propose a nonconvex strongly concave min-max instance-wise formulation for OPAUC and
TPAUC .
In this way, we incorporate the instances selection into the loss calculation to eliminatethe score ranking challenge.
For OPAUC and TPAUC , we employ an efficient stochastic min-maxalgorithm that ensures we can find a ϵ -first order saddle point after O ( ϵ − 3 ) iterations.
Moreover,we present a theoretical analysis of the generalization error of our formulation.
Our conclusionmay contribute to future work about AUC generalization.
Finally, empirical studies over a range oflong-tailed benchmark datasets speak to the effectiveness of our proposed algorithm.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[No] We have
not discovered any negative societal impacts in our study.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No] Our experiments are very time consuming. (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] Because we use the open source software and data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if