Asymptotically Unbiased Instance-wise Regularized Partial AUC Optimization: Theory and Algorithm s
AffiliationAddress
Abstract
The Partial Area Under the ROC Curve (PAUC), typically including One-way
Partial AUC (OPAUC) and Two-way Partial AUC (TPAUC), measures the averageperformance of a binary classifier within a specific false positive rate and/or truepositive rate interval, which is a widely adopted measure when decision constraintsmust be considered.
Consequently, PAUC optimization has naturally attractedincreasing attention in the machine learning community within the last few years.
Nonetheless, most of the existing methods could only optimize PAUC approximately, leading to inevitable biases that are not controllable.
Fortunately, a recentwork presents an unbiased formulation of the PAUC optimization problem viadistributional robust optimization.
However, it is based on the pair-wise formulation of AUC, which suffers from the limited scalability w.r.t.
sample size and aslow convergence rate, especially for TPAUC.
To address this issue, we present asimpler reformulation of the problem in an asymptotically unbiased and instancewise manner.
For both OPAUC and TPAUC, we come to a nonconvex stronglyconcave min-max regularized problem of instance-wise functions.
On top of this,we employ an efficient solver which not only enjoys a linear per-iteration computational complexity w.r.t.
the sample size but also improves the convergence rate for TPAUC optimization from O ( T − 1 / 6 ) to O ( T − 1 / 3 ) ( T is the number of iteration).
Furthermore, we find that the min-max reformulation also facilitates the theoreticalanalysis of generalization error as a byproduct.
Compared with the existing results,we present new error bounds that are much easier to prove and could deal withhypotheses with real-valued outputs.
Finally, extensive experiments on severalbenchmark datasets demonstrate the effectiveness of our method.
Introduction
AUC refers to the Area Under the Receiver Operating Characteristic (ROC) curve [1], where the ROCcurve is obtained by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR) of agiven classifier for all possible thresholds.
Since it is insensitive to the class distribution, AUC hasbecome one of the standard metrics for long-tail, and imbalanced datasets [1, 22, 37].
Consequently,
AUC optimization has attracted increasing attention in the machine learning community ever sincethe early 2000s [13, 5, 34, 17].
Over the last two decades, research on AUC optimization hasevolved from the simplest linear models and decision trees [27, 10, 29, 40] to state-of-the-art deeplearning architectures [21, 14, 36, 42, 41, 32].
With such remarkable success, one can now easilyapply AUC optimization to deal with various real-world problems ranging from financial frauddetection[16, 4, 23], spam email detection [24], to medical diagnosis [24, 37, 36, 42], etc.
However, in such long-tailed applications, we are often interested in a specific region in the ROC curvewhere its area is called Partial AUC (PAUC).
As illustrated in Fig.1, there are two types of PAUC.
[Figure]
FPR
TPR TPR
[Figure]
FPR
TPR
[Figure]
[Table]
Here, One-way Partial AUC (OPAUC) measures the area within an FPR interval ( 0 ≤ FPR ≤ β );while Two-way Partial AUC (TPAUC) measures the area with FPR ≤ β , TPR ≥ α .
Unlikethe full AUC, optimizing PAUC requires selecting top-ranked or/and bottom-ranked instances,leading to a hard combinatorial optimization problem.
Many efforts have been made to solve theproblem [6, 24, 25, 20, 38].
However, the majority of them rely on full-batch optimization andthe approximation of the top (bottom)- k ranking process, which suffers from immeasurable biasesand undesirable efficiency.
Most recently, researchers have started to explore mini-batch PAUCoptimization for deep learning models.
[38] proposed a novel end-to-end optimization frameworkfor PAUC.
This formulation has a fast convergence rate with the help of a stochastic optimizationalgorithm, but the estimation of PAUC is still biased.
Later, [43] proposed a Distributional Robust
Optimization (DRO) framework for PAUC optimization, where the bias can be eliminated by a cleverreformulation and the compositional SGD algorithms [28].
However, they adopt the pair-wise lossfunction which has limited scalability w.r.t.
sample size and O ( ϵ − 4 ) /
O ( ϵ − 6 ) time complexity toreach the ϵ -stationary point for OPAUC / TPAUC .
Considering the efficiency bottleneck comes fromthe pair-wise formulation, we will explore the following question in this paper:
Can we design a simpler, asymptotically unbiased and instance-wise formulation to optimize OPAUC and TPAUC in an efficient way?
To answer this question, we propose an efficient and asymptotically unbiased optimization algorithmfor regularized PAUC maximization with a faster convergence guarantee.
The comparison withprevious results are listed in Tab.1.
We consider both OPAUC and TPAUC maximization, wherefor OPAUC , we focus on maximizing PAUC in the region: FPR ≤ β and for TPAUC we focus onthe region: FPR ≤ β and TPR ≥ α .
We summarize our contributions below.
• With a proper regularization, we propose a nonconvex strongly concave min-max instance-
wise formulation for OPAUC and TPAUC maximization.
On top of our proposed formula-
tion, we employ an efficient stochastic min-max algorithm that finds a ϵ -stationary point
within O ( ϵ − 3 ) iterations.
• We conduct a generalization analysis of our proposed methods.
Our instance-wise reformula- tion can overcome the interdependent issue of the original pair-wise generalization analysis.
The proof is much easier than the existing results.
Moreover, compared with [25, 38], it can be applied to any real-valued hypothesis functions other than the hard-threshold functions. • We conduct extensive experiments on multiple imbalanced image classification tasks.
Theexperimental results speak to the effectiveness of our proposed methods.
Preliminaries
Notations.
In this section, we give some definitions and preliminaries about OPAUC and TPAUC .
Let X ⊆
R
d be the input space, Y = { 0 , 1 } be the label space.
We denote D P and D N as positiveand negative instance distribution, respectively.
Let S
= { z i = ( x i , y i ) } ni =1 be a set of trainingdata drawn from distribution D Z , where n is the number of samples.
Let P ( N resp.)
be a setof positive (negative resp.)
instances in the dataset.
In this paper we only focus on the scoringfunctions f : X (cid:55)→ [0 , 1] .
The output range can be simply implemented by any deep neural networkwith sigmoid outputs.
Standard AUC.
The standard AUC calculates the entire area under the ROC curve.
For mathematicalconvenience, our arguments start with the pair-wise reformulation of AUC.
Specifically, as shownin [1], AUC measures the probability of a positive instance having a higher score than a negativeinstance:
[Equation]
OPAUC .
As mentioned in the introduction, instead of considering the entire region of ROC, wefocus on two forms of PAUC, namely TPAUC and OPAUC.
According to [6], OPAUC is equivalentto the probability of a positive instance x being scored higher than a negative instance x ′ within thespecific range f ( x ′ ) ∈ [ η β ( f ) , 1] s . t .
Pr x ′ ∼D N [ f
[Equation]
x
[Equation]
≥ η β ] =
β :
[Equation]
Practically, we do not know the exact data distributions D P , D N to calculate (2).
Therefore, weturn to the empirical estimations of (2).
Given a finite dataset S with n instances, let n + , n − be thenumbers of positive/negative instances, respectively.
For the OPAUC , its empirical estimation couldbe expressed as [24]:
[Equation]
where n β − = ⌊ n − · β ⌋ ; x ′ [ j ] denotes the j -th largest score among negative samples; ℓ 0 , 1 ( t ) is the 0 − 1loss, which returns 1 if t < 0 and 0 otherwise.
TPAUC .
More recently, [35] argued that an efficient classifier should have low FPR and high
TPR simultaneously.
Therefore, we also study a more general variant called Two-way Partial AUC( TPAUC ), where the restricted regions satisfy TPR ≥ α and FPR ≤ β .
Similar to OPAUC ,
TPAUC measures the probability that a positive instance x ranks higher than a negative instance x ′where f ( x ) ∈ [0 , η α ( f )] , s . t . Pr x ∼D P [ f ( x ) ≤ η α ] = α, f ( x ′ ) ∈
[ η β ( f ) , 1] s .
t . Pr x ′ ∼D N [ f ( x ′ ) ≥ η β ] =β . [Equation]
Similarly to OPAUC , for the TPAUC , we adopt its empirical estimation [35, 38]:
[Equation]
n [Equation] where n α + = ⌊ n + · α ⌋ , and x [ i ] is i -th smallest score among all positive instances.
Problem Formulation
In this section, we introduce how to optimize OPAUC and TPAUC in an asymptotically unbiasedinstance-wise manner.
Note that (3) and (5) are hard to optimize since it is complicated to determinethe positive quantile η α ( f ) and the negative quantile η β ( f ) .
So we can not obtain the bottom-rankedpositive instances and top-ranked negative instances directly.
In this section, we will elaborate onhow to tackle these challenges.
3.1 Optimizing the OPAUC
[Figure] According to Eq.(3), given a surrogate loss ℓ and the finite dataset S , maximizing OPAUC andˆAUC β ( f, S ) is equivalent to solving the following problems, respectively: [Equation]
[Equation]
Step 1: Instance-wise Reformulation.
Here, to simplify the reformulation, we will use the mostpopular squared loss ℓ ( x ) =
(1 − x ) 2 .
Under this setting, the following theorem shows an instancewise reformulation of the OPAUC optimization problem (please see Appendix.
E for the proof):
Theorem 1.
Assuming that f ( x ) ∈ [0 , 1] , ∀ x ∈ X , F op ( f, a, b, γ, t, z ) is defined as:
[Equation]
we have the following conclusions:
(a) ( Population Version .)
We have:
[Equation]
where y = 1 for positive instances, y = 0 for negative instances, Ω γ is the constraint set of γand η
[Equation]
(b) ( Empirical Version .)
Moreover, given a training dataset S with sample size n , denote:
[Equation]
where ˆ η β ( f ) is the empirical quantile of the negative instances in S .
We have:
[Equation]
Step 2: Differentiable Sample Selection.
Thm.1 provides a support to convert the pair-wise lossinto instance-wise loss for OPAUC .
However, the min-max problem Eq.(10) is still difficult to solvedue to the operation I f ( x ′ j ) ≥ η β ( f ) , which requires selecting top-ranked negative instances.
To makethe sample selection process differentiable, we adopt the following lemma.
Lemma 1.
(Lemma 1 [9]) (cid:80) ki
=1 x [ i ] is a convex function of ( x 1 , · · · , x n ) where x [ i ] is the topi element of a set { x 1 , x 2 , · · · , x n } .
Furthermore, for x i , i = 1 , · · · , n , we have (cid:80) ki
=1 x [ i ] =
min s { ks + (cid:80) ni
=1 [ x i − s ] + } , where [ a ] + = max { 0 , a }
[Equation]
Lem.1 proposes an Average Top- k (ATk) loss which is the surrogate loss for top- k loss to eliminatethe sorting problem.
Optimizing the ATk loss is equivalent to selecting top-ranked instances.
Actually,for R β ( f ) , we can just reformulate it as an Average Top- k (ATk) loss.
Denote ℓ − ( x ′ ) =
( f ( x ′ ) −b ) [-
-]2 + 2(1 + γ ) f
( x ′ ) .
In the proof of the next theorem, we will show that ℓ − ( x ′ ) is an increasingfunction w.r.t.
f ( x ′ ) , namely:
[Equation]
where ℓ − ( x ′ )
[ j ] is the j -th largest loss.
And the similar result holds for ˆ R β ( f, S ) .
Then, we canreach to Thm.2 (please see Appendix.
E for the proof):
Theorem 2.
Assuming that f ( x ) ∈ [0 , 1] , for all x ∈ X , we have the equivalent optimization for
OPAUC :
[Equation]
where Ω γ =
[ b − 1 , 1] , Ω s ′ =
[0 , 5] and
[Equation]
[Equation]
[Equation]
[Equation]
Step 3: Asymptotically Unbiased Smoothing.
Even with Thm.2, it is hard to optimize the minmax-min formulation in (13).
A solution is to swap the order max γ and min s ′ to reformulate it as amin-max problem.
The key obstacle to this idea is the non-smooth function [ · ] + .
To avoid the [ · ] + ,we apply the softplus function [12]:
[Equation]
as a smooth surrogate.
It is easy to show that r κ ( x ) κ →∞ → [ x ] + .
Denote G κop ( f, a, b, γ, z , s ′ ) thesurrogate objective where the [ · ] + in G op ( f, a, b, γ, z , s ′ ) is replaced with r κ ( · ) .
We then proceedto solve the surrogate problem:
[Equation]
respectively for the population and empirical version.
Step 4: The Regularized Problem.
It is easy to check that r κ ( x ) has a bounded second-order derivation.
In this way, we can regard G κop ( f, a, b, γ, z , ˆ η β ( f ) , s ′ ) as a weakly-concave function [3] (pleasesee Appendix. D for the definition) of γ .
By employing an ℓ 2 regularization, we turn to a regularizedform:
[Equation]
With a sufficiently large ω , G κ,ωop ( f, a, b, γ, z , s ′ ) is strongly-concave w.r.t.
γ when all the othervariables are fixed.
We then reach a min-max problem in the final step.
Step 5: Min-Max Swapping.
According to min-max theorem [3], if we replace F κop ( f, a, b, γ, z , s ′ )with G κ,ωop ( f, a, b, γ, z , s ′ ) , the surrogate optimization problem satisfies:
[Equation]
where G κ,ωop = G κ,ωop ( f, a, b, γ, z , s ′ )
In this sense, we come to a regularized non-convex stronglyconcave problem.
In Sec.4, we will employ an efficient solver to optimize the parameters.
3.2 Optimizing the TPAUC
According to (5), given a feasible surrogate loss ℓ and finite dataset S , maximizing TPAUC andˆAUC α,β ( f, S ) is equivalent to solving the following problems, respectively:
[Equation] [Equation]
Due to the limited space, we present the result directly, please refer to Appendix.
B for more details.
Similar to OPAUC , we apply the function r κ ( x ) , regularization ωγ 2 and min-max theorem to solvethe problem.
In this sense, we can use
[Equation]
and
[Equation]
to minimize R β,α ( f ) , and ˆ R β,α ( f ) , respectively.
Here:
[Equation]
Training Algorithm
According to the derivations in the previous sections, our goal is then to solve the resulting empiricalmin-max optimization problems in Eq.(17) and Eq.(21).
It is easy to check that they are stronglyconcave w.r.t γ whenever κ ≤ 2 + ω , when f ( x ) , a, b ∈ [0 , 1] , γ ∈ Ω γ , s, s ′ ∈
Ω s , Ω s ′ .
Therefore,we can adopt the nonconvex strongly concave min-max optimization algorithms to solve this problem[15].
In this section, following the work [15], we employ an accelerated stochastic gradient descentascent (ASGDA) method to solve the min-max optimization problem.
We denote θ ∈ R d as theparameters of function f , τ = { θ , a, b, s, s ′ } ∈ Ω τ as the variables for the outer min-problem, whereΩ τ is the closed set.
Algorithm 1 shows the framework of our ASGDA algorithm.
There are twokey steps.
At Line 5-6 of Algorithm 1, variables τ t +1 and γ t +1 are updated in a momentum way.
Moreover, the convex combination ensures that they are always feasible given that the initial solutionis feasible.
At Line 9-10 , using the momentum-based variance reduced technique, we can estimatethe stochastic first-order partial gradients v t and w t in a more stable manner. [Equation] [Equation] [Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
With the following smoothness assumption, we show the convergence rate in Thm.3.
Assumption 1.
G ( · ) ( τ , γ ; B ) has Lipschitz continuous gradients, i.e., there is a positive scalar L Gsuch that for any τ ,
τ ′ ∈
Ω τ , γ, γ ′ ∈ Ω γ ,
[Equation]
[Equation]
Theorem 3.
(Theorem 9 [15])
Supposing that Asm.1 holds, let { τ t , γ t } be a sequence generated byour method, if the learning rate satisfies:
[Equation]
Then we have:
[Equation]
where ∥ 1 ν ( τ t − P Ω τ ( τ t − ν ∇ F ( · ) ( τ t )))
∥ is the l 2 -norm of gradient mapping metric for the outerproblem [7, 11, 30] with F ( · ) ( τ t ) =
max γ ∈ Ω γ G ( · ) ( τ t , γ ) .
When b = 1 , it is easy to verify thatk =
O (1) , λ
= O ( µ ) , ν − 1 =
O ( L G /µ ) , c
1 = O (1) , c
2 =
O ( L 2 G ) and m =
O ( L 6 G ) .
Then we have
M ′′ =
O ( L 3 G /µ 3 ) .
Thus, the algorithm has a convergence rate of O ( ( L G /µ ) 3 / 2 T 1 /) .
By ( L G /µ ) 3 / 2 T 1 /≤ ϵ ,then the iteration number to achieve ϵ -first-order saddle point which satisfies: T ≥ ( L G /µ ) 4 .
5 ϵ − 3 .  Generalization Analysis In this section, we theoretically analyze the generalization performance of our proposed estimatorsfor OPAUC (please see the Appendix.
F for the TPAUC ).
According to Thm.2 in Sec.3, we knowthat the generalization error of OPAUC with a surrogate loss ℓ can be measured as:
[Equation]
and
[Equation]
Following the ERM paradigm, to prove the uniform convergence result over a hypothesis class F ofthe scoring function f , we need to show that:
[Equation]
According to the aforementioned discussion, we only need to prove that:
[Equation]
To prove this, we need to define the measure of the complexity of the class F .
Here we adopt the
Radermacher complexity ℜ as in [2] (please see Appendix.
D for the definition).
Specifically, wecome to the following definition:
Definition 1.
The empirical Rademacher complexity of positive and negative instances with respectto S is defined as:
[Equation]
where ( σ 1 , · · · , σ m ) are independent uniform random variables taking values in {− 1 , +1 } .
Finally, we come to the generalization bound as follows:
Theorem 4.
For any δ > 0 , with probability at least 1 − δ over the draw of an i.i.d.
sample set S ofsize m , for all f ∈ F we have:
[Equation]
Remark 1.
Although the results we obtain are similar to some previous studies.
[38, 25], ourgeneralization analysis is simpler and does not require complex error decomposition.
Moreover,our results hold for all real-valued hypothesis class with outputs in [0 , 1] , while the previous results[38, 25] only hold for hard-threshold functions.
Experiment
In this section, we conduct a series of experiments on different datasets for both OPAUC and
TPAUC optimization.
Due to space limitations, please refer to the Appendix.
C for the details ofimplementation and competitors.
6.1 Setups
We adopt three imbalanced binary classification datasets: CIFAR-10-LT [8], CIFAR-100-LT [19] and
Tiny-ImgaeNet-200-LT following the instructions in [38], where the binary datasets are constructedby selecting one super category as positive class and the other categories as negative class.
Please see
Appendix. C for more details.
The evaluation metrics in experiments are ˆAUC β and ˆAUC α,β .
6.2 Overall Results
In Tab.2 and Tab.3, we record the performance on test sets of all the methods on three subsets of
CIFAR-10-LT, CIFAR-100-LT, and Tiny-Imagent-200-LT.
Each method is tuned independently for
OPAUC and TPAUC metrics.
From the results, we make the following remarks: (1) Our proposedmethods outperform all the competitors on all the datasets in most cases, except for the CIFAR-100LT Subset 3.
Even for failure cases, our methods attain fairly competitive results compared with thecompetitors for OPAUC and TPAUC .
In addition, we can see that the normal AUC optimizationmethod AUC-M has poor performance under PAUC metric.
This demonstrates the necessity ofdeveloping the PAUC optimization algorithm.
(3) Approximation methods like DROS, AUC-poly,and AUC-exp have lower performance than the unbiased algorithms.
Above all, the empirical resultssuggest the effectiveness of our proposed method.
[Table]
6.3 Convergence Analysis
In the convergence experiments, for sake of fairness [Table] we did not use warm-up.
All algorithms usehyperparameters in the performance experiments.
{+We show the plots of training convergence in Fig.
+}
[Table] [Figure] [Figure] and Fig.3 on CIFAR-10 for both OPAUC and TPAUC .
Due to the space limitation, the other resultscould be found in Appendix.
C.
According to the figures, we can make the following observations: (1)
Our algorithm and DROE converge faster than other methods for OPAUC .
However, for TPAUCoptimization, the DROE converges very slowly due to its complicated algorithm, while our methodstill shows the best convergence property in most cases.
(2) It’s notable that our algorithm convergesto stabilize after twenty epochs in most cases.
That means our method has better stability in practice.
Conclusion
In this paper, we focus on designing an efficient and asymptotically unbiased algorithm for PAUC.
We propose a nonconvex strongly concave min-max instance-wise formulation for OPAUC and
TPAUC .
In this way, we incorporate the instances selection into the loss calculation to eliminatethe score ranking challenge.
For OPAUC and TPAUC , we employ an efficient stochastic min-maxalgorithm that ensures we can find a ϵ -first order saddle point after O ( ϵ − 3 ) iterations.
Moreover,we present a theoretical analysis of the generalization error of our formulation.
Our conclusionmay contribute to future work about AUC generalization.
Finally, empirical studies over a range oflong-tailed benchmark datasets speak to the effectiveness of our proposed algorithm.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[No] We have
not discovered any negative societal impacts in our study.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[No] Our experiments are very time consuming.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?