Large-scale Optimization of Partial AUC in a Range of False Positive Rates
s
AffiliationAddress
Abstract
The area under the ROC curve (AUC) is one of the most widely used performancemeasures for classification models in machine learning.
However, it summarizesthe true positive rates (TPRs) over all false positive rates (FPRs) in the ROC space,which may include the FPRs with no practical relevance in some applications.
Thepartial AUC, as a generalization of the AUC, summarizes only the TPRs over aspecific range of the FPRs and is thus a more suitable performance measure inmany real-world situations.
Although partial AUC optimization in a range of FPRshad been studied, existing algorithms are not scalable to big data and not applicableto deep learning.
To address this challenge, we cast the problem into a non-smoothdifference-of-convex (DC) program for any smooth predictive functions (e.g.,deep neural networks), which allowed us to develop an efficient approximatedgradient descent method based on the Moreau envelope smoothing technique,inspired by recent advances in non-smooth DC optimization.
To increase theefficiency of large data processing, we used an efficient stochastic block coordinateupdate in our algorithm.
Our proposed algorithm can also be used to minimizethe sum of ranked range loss, which also lacks efficient solvers.
We establisheda complexity of ˜ O
(1 /ϵ 6 ) for finding a nearly ϵ -critical solution.
Finally, wenumerically demonstrated the effectiveness of our proposed algorithms in trainingboth linear models and deep neural networks for partial AUC maximization andsum of ranked range loss minimization.
Introduction
The area under the receiver operating characteristic (ROC) curve (AUC) is one of the most widelyused performance measures for classifiers in machine learning, especially when the data is imbalancedbetween the classes [7,19].
Typically, the classifier produces a score for each data point.
Then a datapoint is classified as positive if its score is above a chosen threshold; otherwise, it is classified asnegative.
Varying the threshold will change the true positive rate (TPR) and the false positive rate(FPR) of the classifier.
The ROC curve shows the TPR as a function of the FPR that correspondsto the same threshold.
Hence, maximizing the AUC of a classifier is essentially maximizing theclassifier’s average TPR over all FPRs from zero to one.
However, for some applications, some FPRregions have no practical relevance.
So does the TPR over those regions.
For example, in clinicalpractice, a high FPR in diagnostic tests often results in a high monetary cost, so people may onlyneed to maximize the TPR when the FPR is low [13,34,64].
Moreover, since two models with thesame AUC can still have different ROCs, the AUC does not always reflect the true performance of amodel that is needed in a particular production environment [8].
As a generalization of the AUC, the partial AUC (pAUC) only measures the area under the ROCcurve that is restricted between two FPRs.
A probabilistic interpretation of the pAUC can be foundin [13].
In contrast to the AUC, the pAUC represents the average TPR only over a relevant range of FPRs and provides a performance measure that is more aligned with the practical needs in someapplications.
In literature, the existing algorithms for training a classifier by maximizing the pAUC include theboosting method [29] and the cutting plane algorithm [41–43].
However, the former has no theoreticalguarantee, and the latter applies only to linear models.
More importantly, both methods requireprocessing all the data in each iteration and thus, become computationally inefficient for largedatasets.
In this paper, we proposed an approximate gradient method for maximizing the pAUC that worksfor nonlinear models (e.g., deep neural networks) and only needs to process randomly sampledpositive and negative data points of any size in each iteration.
In particular, we formulated themaximization of the pAUC as a non-smooth difference-of-convex (DC) program [30,54].
Due tonon-smoothness, most existing DC optimization algorithms cannot be applied to our formulation.
Motivated by [52], we approximate the two non-smooth convex components in the DC program bytheir Moreau envelopes and obtained a smooth approximation of the problem, which will be solvedusing the gradient descent method.
Since the gradient of the smooth problem cannot be calculatedexplicitly, we approximated the gradient by solving the two proximal-point subproblems defined byeach convex component using the stochastic block coordinate descent (SBCD) method.
Our method,besides its low per-iteration cost, has a rigorous theoretical guarantee, unlike the existing methods.
In fact, we show that our method finds a nearly ϵ -critical point of the pAUC optimization problemin ˜ O
( ϵ − 6 ) iterations with only small samples of positive and negative data points processed periteration.
This is the main contribution of this paper.
Note that, for non-convex non-smooth optimization, the existing stochastic methods [10,11] find annearly ϵ -critical point in O ( ϵ − 4 ) iterations under a weak convexity assumption.
Our method needs
O ( ϵ − 6 ) iterations because our problem is a DC problem with both convex components non-smoothwhich is much more challenging than a weakly non-convex minimization problem.
In addition, ouriteration number matches the known best iteration complexity for non-smooth non-convex min-maxoptimization [33,46] and non-smooth non-convex constrained optimization [35].
In addition to pAUC optimization, our method can be also used to minimize the sum of ranked range(SoRR) loss, which can be viewed as a special case of pAUC optimization.
Many machine learningmodels are trained by minimizing an objective function, which is defined as the sum of losses over alltraining samples [60].
Since the sum of losses weights all samples equally, it is insensitive to samplesfrom minority groups.
Hence, the sum of top- k losses [17,49] is often used as an alternative objectivefunction because it provides the model with robustness to non-typical instances.
However, the sum oftop- k losses can be very sensitive to outliers, especially when k is small.
To address this issue, [24]proposed the SoRR loss as a new learning objective, which is defined as the sum of a consecutivesequence of losses from any range after the losses are sorted.
Compared to the sum of all losses andthe sum of top- k losses, the SoRR loss maintains a model’s robustness to a minority group but alsoreduces the model’s sensitivity to outliers.
See Fig.1 in [24] for an illustration of the benefit of usingthe SoRR loss over other ways of aggregating individual losses.
To minimize the SoRR loss, [24] applied a difference-of-convex algorithm (DCA) [3, 54], whichlinearizes the second convex component and solves the resulting subproblem using the stochasticsubgradient method.
DCA has been well studied in literature; but when the both components arenon-smooth, as in our problem, only asymptotic convergence results are available.
To establish thetotal number of iterations needed to find an ϵ -critical point in a non-asymptotic sense, most existingstudies had to assume that at least one of the components is differentiable, which is not the case inthis paper.
Using the approximate gradient method presented in this paper, one can find a nearlyϵ -critical point of the SoRR loss optimization problem in ˜ O
( ϵ − 6 ) iterations.
Related Works
The pAUC has been studied for decades [26, 37, 57].
However, most studies focused on its estimation [13] and application as a performance measure, while only a few studies were devoted tonumerical algorithms for optimizing the pAUC.
Besides the boosting method [29] and the cutting plane algorithm
[41–43] mentioned in the previous section, [59] and [65] developed surrogate opti- mization techniques that directly maximize a smooth approximation of the pAUC or the two-waypAUC [64].
However, their approaches can only be applied when the FPR starts from exactly zero.
On the contrary, our method allows the FPR to start from any value between zero and one.
[61]and [47] developed algorithms that use the pAUC as a criterion for creating a linear combination ofmultiple existing classifiers while we consider directly train a classifier using the pAUC.
DC optimization has been studied since the 1950s [2,20].
We refer interested readers to [30,45,54,55,58], and the references therein.
The actively studied numerical methods for solving a DC programinclude DCA [3,50,54,55], which is also known as the concave-convex procedure [32,51,68], theproximal DCA [5,38,40,53], and the direct gradient methods [28].
However, when the two convexcomponents are both non-smooth, the existing methods have only asymptotic convergence resultsexcept the method by [1], who considered a stopping criterion different from ours.
When at least onecomponent is smooth, non-asymptotic convergence rates have been established with and without the
Kurdyka-Łojasiewicz (KL) condition [5,6,28,50,62].
The algorithms mentioned above are deterministic and require processing the entire dataset periteration.
Stochastic algorithms that process only a small data sample per iteration have beenstudied [12,21,31,36,44].
However, they all assumed smoothness on at least one of the two convexcomponents in the DC program.
The stochastic methods of [4, 56, 63] can be applied when bothcomponents are non-smooth but their methods require an unbiased stochastic estimation of thegradient and/or value of the two components, which is not available in the DC formulation of thepAUC maximization problem in this paper.
The technique most related to our work is the smoothing method based on the Moreau envelope [16,18,22,23,39,52].
Our work was motivated by [39,52], but the important difference is that they studieddeterministic methods and assumed either that one function is smooth or that the proximal-pointsubproblems can be solved exactly, which we do not assume.
However, [52] developed an algorithmwhen there were linear equality constraints, which we do not consider in this paper.
Finally, it was brought to our attention that a recent work [69] also considered partial AUC maximization with a non-convex objective.
However, the algorithms in [69] can only optimize one-way pAUCwith FPR in a range (0 , β ) , while our algorithm is more general and can be applied to FRP in anyrange ( α, β ) with α ≥ 0 , which is more challenging for α > 0 .
Preliminary
We consider a classical binary classification problem, where the goal is to build a predictive modelthat predicts a binary label y ∈ { 1 , − 1 } based on a feature vector x ∈ R p .
Let h w : R p
→ R be thepredictive model parameterized by a vector w ∈ R d , which produces a score h w ( x ) for x .
Thenx is classified as positive ( y = 1 ) if h w ( x ) is above a chosen threshold and classified as negative( y = − 1 ), otherwise.
Let X + =
{ x + i } N + i and X − = { x − i } N − i be the sets of feature vectors of positive and negativetraining data, respectively.
The problem of learning h w through maximizing its empirical AUC onthe training data can be formulated as
[Equation]
where 1 ( · ) is the indicator function which equals one if the inequality inside the parentheses holds andequals zero, otherwise.
According to the introduction, pAUC can be a better performance measure ofh w than AUC.
Consider two FPRs α and β with 0 ≤ α < β ≤ 1 .
For simplicity of exposition, weassume N − α and N − β are both integers.
Let m =
N − α and n = N − β .
The problem of maximizingthe empirical pAUC with FPR between α and β can be formulated as
[Equation]
where [ j ] denotes the index of the j th largest coordinate in vector ( h w ( x − j ))
N − j =1 with ties brokenarbitrarily.
Note that N + ( n − m ) in (2) is a normalizer that makes the objective value between zero and one.
Solving (2) is challenging due to discontinuity.
Let ℓ : R → R be a differentialnon-increasing loss function.
Problem (2) can be approximated by the loss minimization problem
[Equation]
To facilitate the discussion, we first introduce a few notations.
Given a vector S
=
( s i )
Ni =1 ∈ R N andan integer l with 0 ≤ l ≤ N , the sum of the top- l values in S is
[Equation]
where [ j ] denotes the index of the j th largest coordinate in S with ties broken arbitrarily.
For integers l 1 and l 2 with 0 ≤ l 1 < l 2 ≤ N , ϕ l 2 ( S ) − ϕ l 1 ( S ) is the sum from the ( l 1 +1)
th to the l 2 th (inclusive) largest coordinates of S , also called a sum of ranked range (SoRR).
In addition, we define vectors
[Equation]
for i = 1 , . . .
, N + , where s ij ( w ) :=
ℓ ( h w ( x + i ) − h w ( x − j )) for i
= 1 , . .
. , N + and j = 1 , . .
. , N − .
Since ℓ is non-increasing, the j th largest coordinate of S i ( w ) is ℓ ( h w ( x + i ) − h w ( x − [ j ] )) .
As a result,we have, for i = 1 , . . .
, N + ,
[Equation]
Hence, after dropping the normalizer, (3) can be equivalently written as
[Equation]
) where
[Equation]
Next, we introduce an interesting special case of (5), namely, the problem of minimizing SoRR loss.
We still consider a supervised learning problem but the target y ∈ R does not need to be binary.
We want to predict y based on a feature vector x ∈ R p using h w ( x ) .
With a little abuse of notation, we measure the discrepancy between h w ( x ) and y by ℓ ( h w ( x ) , y ) , where ℓ : R 2 → R + is a loss function.
We consider learning the model’s parameter w from a training set D = { ( x j , y j ) } Nj =1 , where x j ∈ R p and y j ∈ R for j = 1 , . . .
, N , by minimizing the SoRR loss.
More specifically, we define vector
[Equation]
where s j ( w ) : =
ℓ ( h w ( x j ) , y j ) , j = 1 , . . . , N .
Recall (4).
For any integers m and n with 0 ≤ m <n ≤ N , the problem of minimizing the SoRR loss with a range from m + 1 to n is formulated asmin w { ϕ n ( S ( w ))
− ϕ m ( S ( w )) } , which is an instance of (5) withf l = ϕ l ( S ( w )) for l = m, n.
If we view S i ( w ) and S ( w ) only as vector-value functions of w but ignore how they are formulatedusing data, (7) is a special case of (6) with N + = 1 and N − = N .
Nearly Critical Point and Moreau Envelope Smoothing
We first develop a stochastic algorithm for (5) with f l defined in (6).
To do so, we make the followingassumptions, which are satisfied by many smooth h w ’s and ℓ ’s.
Assumption 1 (a) s ij ( w ) is smooth and there exists L ≥
such that 2 ∥∇ s ij ( w ) − ∇ s ij ( v ) ∥ ≤
L ∥ w − v ∥ for any w , v ∈ R d , i = 1 , . .
. , N + and j = 1 , . .
. , N − .
(b) There exists B ≥
such that∥∇ s ij ( w ) ∥ ≤ B for any w ∈ R d , i = 1 , . . . , N + and j = 1 , . . . , N − .
(c) F ∗ > −∞ .
Given f : R d → R ∪ { + ∞} , the subdifferential of f is
[Equation]
where each element in ∂f ( w ) is called a subgradient of f at w .
We say f is ρ -weakly convex forsome ρ ≥ 0 if f ( v ) ≥ f ( w ) + ⟨ ξ , v − w ⟩ − ρ 2 ∥ v − w ∥ 2 for any v and w and ξ ∈ ∂f ( w ) andsay f is ρ -strongly convex for some ρ ≥
if f ( v ) ≥ f ( w ) + ⟨ ξ , v − w ⟩ + ρ 2 ∥ v − w ∥ 2 for anyv and w and ξ ∈ ∂f ( w ) .
It is known that, if f is ρ -weakly convex, then f ( w ) + 12 µ ∥ w ∥ 2 is a( µ − 1 − ρ ) -strongly convex function when µ − 1 > ρ .
Under Assumption 1, ϕ l ( S i ( w )) is a composite of the closed convex function ϕ l and the smooth map
S i ( w ) .
According to Lemma 4.2 in [14], we have the following lemma.
Lemma 1 Under Assumption 1, f m ( w ) and f n ( w ) in (6) are ρ -weakly convex with ρ := N + N − L .
To solve (5) numerically, we need to overcome the following challenges.
(i) F ( w ) is non-convexeven if each s ij ( w ) is convex.
In fact, F ( w ) is a DC function because, by Lemma 1, we can represent
F ( w ) as the difference of the convex functions f n ( w ) + 12 µ ∥ w ∥ 2 and f m ( w ) + 12 µ ∥ w ∥ 2 withµ − 1 > ρ .
(ii) F ( w ) is non-smooth due to ϕ l so that finding an approximate critical point (definedbelow) of F ( w ) is difficult.
(iii)
Computing the exact subgradient of f l ( w ) for l = m, n requiresprocessing N + N − data pairs, which is computationally expensive for a large data set.
Because of challenges (i) and (ii), we have to consider a reasonable goal when solving (5).
We sayw ∈ R d is a critical point of (5) if 0 ∈ ∂f n ( w ) − ∂f m ( w ) .
Given ϵ > 0 , we say w ∈ R d is anϵ -critical point of (5) if there exists ξ ∈ ∂f n ( w ) − ∂f m ( w )
such that ∥ ξ ∥ ≤ ϵ .
A critical pointcan only be achieved asymptotically in general.
Within finitely many iterations, there also existsno algorithm that can find an ϵ -critical point unless at least one of f m and f n is smooth, e.g., [63].
Since f m and f n are both non-smooth, we have to consider a weaker but achievable target, which isa nearly ϵ -critical point defined below.
Definition 1 Given ϵ > 0 , we say w ∈ R d is a nearly ϵ -critical point of (5) if there exist ξ , w ′ ,
andw ′′ ∈ R d such that ξ ∈ ∂f n ( w ′ )
− ∂f m ( w ′′ ) and max {∥ ξ ∥ , ∥ w − w ′ ∥ , ∥ w − w ′′ ∥} ≤ ϵ .
Definition 1 is reduced to the ϵ -stationary point defined by [39,52] when w equals w ′ or w ′′ .
However,obtaining their ϵ -stationary point requires exactly solving the proximal mapping of f m or f n whilefinding a nearly ϵ -critical point requires only solving the proximal mapping inexactly.
When w isgenerated by a stochastic algorithm, we also call w a nearly ϵ -critical point if it satisfies Definition 1with each ∥ · ∥ replaced by E ∥ · ∥ .
Motivated by [52] and [39], we approximate non-smooth F ( w ) by a smooth function using the
Moreau envelopes.
Given a proper, ρ -weakly convex and closed function f on R d , the Moreauenvelope of f with the smoothing parameter µ ∈ (0 , ρ − 1 ) is defined as
[Equation]
and the proximal mapping of f is defined as
[Equation]
Note that the v µf ( w ) is unique because the minimization above is strongly convex.
Standard resultsshow that f µ ( w ) is smooth with ∇ f µ ( w ) = µ − 1 ( w − v µf ( w )) and v µf ( w ) is (1 − µρ ) − 1 -Lipschitzcontinuous.
See Proposition 13.37 in [48] and Proposition 1 in [52].
Hence, using the Moreauenvelope, we can construct a smooth approximation of (5) as follows
[Equation]
Function F µ has the following properties.
The first property is shown in [52].
We give the proof forthe second in Appendix B.
Lemma 2 Suppose Assumption 1 holds and µ > ρ − 1 with ρ defined in Lemma 1.
The followingclaims hold
∇ F µ ( w ) = µ − 1 ( v µf m ( w ) − v µf n ( w )) and it is L µ -Lipschitz continuous with L µ = 2 µ − µ 2 ρ .2.
If ¯ v and w are two random vectors such that E ∥∇ F µ ( w ) ∥ 2 ≤ min { 1 , µ − 2 } ϵ 2 / 4 and E ∥ ¯ v −v µf l ( w ) ∥ 2 ≤ ϵ 2 / 4 for either l = m or l = n , then ¯ v is a nearly ϵ -critical points of (5) .
Since F µ is smooth, we can directly apply a first-order method for smooth non-convex optimizationto (10).
To do so, we need to evaluate ∇ F µ ( w ) , which requires computing v µf m ( w ) and v µf n ( w ) ,i.e., exactly solving (9) with f = f m and f = f n , respectively.
Computing the subgradients of f mand f n require processing N + N − data pairs which is costly.
Unfortunately, the standard approachof sampling over data pairs does not produce unbiased stochastic subgradients of f m and f n due tothe composite structure ϕ l ( S i ( w )) .
In the next section, we will discuss a solution to overcome thischallenge and approximate v µf m ( w ) and v µf n ( w ) , which leads to an efficient algorithm for (10).
Algorithm for pAUC Optimization
Consider (10) with f l defined in (6) for l = m and n .
To avoid of processing N + N − data points, onemethod is to introduce dual variables p
i =
( p ij )
N − j =1 for i = 1 , . . .
, N +
and formulate f l asf
[Equation]
where P l = { p ∈ R N − | (cid:80) N − j =1 p j = l, p j ∈ [0 , 1] } .
Then (10) can be reformulated as a minmax problem and solved by a primal-dual stochastic gradient method (e.g. [46]).
However, themaximization in (11) involves N + N − decision variables and equality constraints, so the per-iterationcost is still O
( N + N − ) even after using stochastic gradients.
To further reduce the per-iteration cost, we take the dual form of the maximization in (11) (see
Lemma 4 in Appendix B) and formulate f l as
[Equation]
where λ = ( λ 1 , . .
, λ N + ) .
Hence, (9) with f = f l for l = m and n can be reformulated as
[Equation]
Note that g l ( v , λ ) is jointly convex in v and λ when µ − 1 > ρ = N + N − L (see Lemma 3 in
Appendix B).
Thanks to formulation (13), we can construct stochastic subgradient of g l and applycoordinate update to λ by sampling indexes i ’s and j ’s, which significantly reduce the computationalcost
when N +
and N − are both large.
We present this standard stochastic block coordinate descent(SBCD) method for solving (13) in Algorithm 1 and present its convergence property as follows.
Proposition 1 Suppose Assumption 1 holds and µ − 1 > ρ = N + N − L , θ t = dist ( λ (0) , Λ ∗ ) √ ITN − andη t = ∥ v µfl ( w ) − w ∥ N + N − B √ T for any t in Algorithm 1.
It holds that
[Equation] [Equation]
where Λ ∗ = arg min λ g l ( v µf l ( w ) , λ ) .
Using Algorithm 1 to compute an approximation of v µf l ( w ) for l = m and n and thus, an approximation of ∇ F µ ( w ) , we can apply an approximate gradient descent (AGD) method to (10) and find anearly ϵ -critical point of (5) according to Lemma 2.
We present the AGD method in Algorithm 2 andits convergence property as follows.
2: Set choose θ t
− 1 t =0 .
3: for
− 1 do 4:
Sample I t ⊂ { 1 , . .
+ with
| = and sample J
⊂ { 1 , . . .
, N − } with |J t | =
J .
Compute stochastic subgradient w.r.t.
v :
[Equation]
(cid:80)
[Equation]
Proximal subgradient update on v :
[Equation]
=
[Equation]
Compute stochastic subgradient w.r.t.
λ i for i ∈
I t :
[Equation] [Equation]
t
[Equation]
> λ ( t
[Equation]
(cid:17) for i ∈
I t 8:
Stochastic block subgradient update on λ
i for i ∈
I t : λ ( t +
[Equation]
i = λ ( t
[Equation]
i − θ t G ( t
[Equation]
i for i ∈
I t and λ ( t +1) i
= λ ( t ) i for i / ∈
I t .
(15) 9: end for 10: Output: (¯ v , ¯ λ ) =
1 T (cid:80) T − 1 t =
( v ( t ) , λ ( t ) ) .
[Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation]
Theorem 1 Suppose Assumption 1 holds and Algorithm 1 is called in iteration k of Algorithm 2 with parameters µ − 1 > ρ
= N + N − L , θ t = dist (¯ λ ( k ) , Λ ∗ k )
√ IT k N − , η t = ∥ v µfl ( w ( k ) ) − w ( k ) ∥
N + N − B √ T k for any t , and
T k = max (cid:26) 144 N 2 + N 2 − D 2 l ( k + 1)
I ( µ − 1 − ρ ), 4 N 2 + N 2 − µ 2 l 2 B 2 ( k +
[Equation]
( µ − 1 − ρ ), 6 µl 2 B
[Equation]
k + 1) 2( µ − 1 − ρ )
[Equation]
(cid:27) where Λ ∗ k = arg min λ g l ( v µf l ( w ( k ) ) , λ ) and
D l := max (cid:26) dist (¯ λ (0) l , Λ ∗ 0 ) ,  (cid:18)
[Equation]
Then ¯ v (¯ k ) n is a nearly ϵ -critical point of (5) with f l defined in (6) with K no more than
[Equation]
According to Theorem 1. To find a nearly ϵ -critical point of (5), we need K =
˜
O
( ϵ − 2 ) iterations in
Algorithm 2 and (cid:80) K − 1 k =
T k
= O
( K 3 )
=
˜ O
( ϵ − 6 ) iterations of Algorithm 1 in total across all calls.
The technique in the previous sections can be directly applied to minimize the SoRR loss, which isformulated as (5) but with f l defined in (7). Due to the limit of space, we present the algorithm forminimizing the SoRR loss and its convergence result in Appendix A.
[Figure]
[-150 # of epoch 1.52.
-][Figure]
T r a i n i ng
Lo ss
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
epoch 0.
[Figure]
p AUC on T r a i n i ng se t
[Figure]
[-˜ O ( ϵ − 2 ) iterations and the total complexity is (cid:80) K − 1 k =
 T k .-]_D
[-− w ( k ) ∥ 2 are O (1)(see (27) and (30) in Appendix D) which ensures that T k = Θ( k 2 ) and thus the total complexity is 238 (cid:80) K −k =
-]
[Figure]
Numerical Experiments
In this section, we demonstrate the effectiveness of our algorithm AGD-SBCD for pAUC maximization and SoRR loss minimization problems.
All experiments are conducted in Python and Matlab ona computer with the CPU 2GHz Quad-Core Intel Core i5 and the GPU NVIDIA GeForce RTX {+
+} Ti.
All datasets we used are publicly available and contain no personally identifiable information andoffensive contents.
6.1 Partial AUC Maximization
For maximizing pAUC, we focus on large-scale imbalanced medical dataset CheXpert [25], which islicensed under CC-BY-SA and has 224,316 images.
We construct five binary classification tasks withthe logistic loss ℓ ( z )
=
log(1 + exp( − z )) for predicting five popular diseases, Cardiomegaly (D1),
Edema (D2), Consolidation (D3), Atelectasis (D4), and P. Effusion (D5).
For comparison of training convergence, we consider different methods for optimizing the partial
AUC.
We compare with two baselines, DCA [24] (see Appendix E.3 for details) and SVM pAUCtight [42].
Since DCA and SVM pAUC -tight cannot be applied to deep neural networks, we focus onlinear model and use a pre-trained deep neural network to extract a fixed dimensional feature vectorsof 1024 .
The deep neural network was trained by optimizing the cross-entropy loss following thesame setting as in [66].
For AGD-SBCD, we fix the I = J = 100 , m = 10 4 and n = 10 5 . In the spirit of Theorem 1,within Algorithm 3 called in the k th main iteration, η t is set to c ( k +1)
N + N − , θ t is set to c ( k +1)
N −for any t with c tuned in { 0 . 1 , 1 , 5 , 10 , 20 , 30 } and T k is set to 50( k + 1) 2 . Parameters µ is chosenfrom { 10 2 , 10 3 } / ( N + N − ) and γ is tuned from { 2 × 10 3 , 5 × 10 3 } / ( N + N − ) . For DCA, we onlyimplement the setting of DCA.Changing.lr (see Appendix E.1 for descriptions). In Algorithm 4called in the k th main iteration, η t and θ t are selected following the same process as in AGD-SBCDand T is set to C ( k + 1) 2 with C chosen from { 100 , 200 , 500 } .
For the SVM pAUC -tight method,we use their MATLAB code in [42] and tune hyper-parameter C from { 10 − 3 , 10 − 2 , 10 − 1 , 10 0 } .
In Figure 1 and Appendix E.2 Figure 3, we show how the training loss (the objective value of (3)) andnormalized partial AUC on the training data change with the number of epochs.
We observe that forall of these five diseases, our algorithm converges much faster than DCA and we get a better partial AUC than DCA.
The comparison between our AGD-SBCD and SVM pAUC -tight on training data are shown in Table1.
As shown from the second to the fifth column of Table 1, our algorithm needs only a few secondsto exceed the pAUCs that SVM pAUC -tight takes more than one minute to return.
As shown fromsixth to eighth column, our algorithm eventually improves the pAUC by at least 12% compared with SVM pAUC -tight.
DCA is not included in the tables because it computes deterministic subgradients,which leads to a runtime significantly longer than the other two methods.
To compare the testing performances, we consider the deep neural networks besides the linear model.
For linear model, we still compare with DCA and SVM pAUC -tight.
For deep neural networks,we compare with the naive mini-batch based method (MB) [27] and methods based on differentoptimization objectives, including the cross-entropy loss (CE) and the AUC min-max margin loss(AUC-M) [67].
We learn the model DenseNet121 from scratch with the CheXpert training data splitin
train/val=9:1 and the CheXpert validation dataset as the testing set, which has 234 samples.
The [Table] range of FPRs in pAUC is [0.05, 0.5].
For optimizing CE, we use the standard Adam optimizer.
For optimizing AUC-M, we use the PESG optimizer in [67].
We run each method 10 epochsand the learning rate ( c in AGD-SBCD) of all methods is tuned from { 10 − 5 ∼ 10 0 } .
The minibatch size is 32.
For AGD-SBCD, T k is set to 50( k + 1) 2 , µ is set to 10 3 N + N − and γ is tuned from{ 0 . 1 , 1 , 2 } × 10 3 / ( N + N − ) .
For MB, the learning rate decays in the same way as in [27].
For
CE and AUC-M, the learning rate decays 10-fold after every 5 epochs.
For AUC-M, we tune thehyperparameter γ in {100, 500, 1000}.
For each method, the validation set is used to tune thehyperparameters and select the best model across all iterations.
The results of the pAUCs on thetesting set are reported in Table 2, which shows that our algorithm performs the best for all diseases.
The complete ROC curves on the testing set are shown in Appendix E.2.
[Table]
6.2 SoRR Loss Minimization
For SoRR loss minimization, we compare with the baseline DCA.
We focus on learning a linearmodel h w ( x ) =
w T x with four benchmark datasets from the UCI [15] data repository preprocessedby Libsvm [9]: a9a, w8a, ijcnn1 and covtype.
The statistics of these datasets are summarized in Table3 in Appendix E.1. We use logistic loss l ( h w ( x ) , y ) = − y log ( h w ( x )) − (1 − y )
log(1 − h w ( x ))where label y ∈ { 0 , 1 } .
Due to the limit space, we present the process of tuning hyperparameters andthe convergence curves (Figure2) in Appendix E.1.
From the curves, we notice that our algorithmreduce SoRR loss faster than DCAs for all of these four datasets.
Conclusion
Most existing methods for optimizing pAUC are deterministic and only have an asymptotic convergence property.
We formulate pAUC optimization as a non-smooth DC program and developa stochastic subgradient method based on the Moreau envelope smoothing technique.
We showthat our method finds a nearly ϵ -critical point in ˜ O
( ϵ − 6 ) iterations and demonstrate its performancenumerically.
A limitation of this paper is the smoothness assumption on s ij ( w ) , which does not holdfor some models, e.g., neural networks using ReLU activation functions.
It is a future work to extendour results for non-smooth models.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Algorithm 1 and 2 and Section 6.
(b) Did you describe the limitations of your work?
[Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[No] We are notaware of any negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Section 6.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 6.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 6.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 6.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite them
as [9,15,25]
(b) Did you mention the license of the assets?
[Yes] See Section 6.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating? [N/A] We only use publicly available data. (e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] See Section 6.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]