Large-scale Optimization of Partial AUC in a Range of False Positive Rates
Abstract
The area under the ROC curve (AUC) is one of the most widely used performance measures for classification models in machine learning.
However, it summarizes the true positive rates (TPRs) over all false positive rates (FPRs) in the ROC space, which may include the FPRs with no practical relevance in some applications.
The partial AUC, as a generalization of the AUC, summarizes only the TPRs over a specific range of the FPRs and is thus a more suitable performance measure in many real-world situations.
Although partial AUC optimization in a range of FPRs had been studied, existing algorithms are not scalable to big data and not applicable to deep learning.
To address this challenge, we cast the problem into a non-smooth difference-of-convex (DC) program for any smooth predictive functions (e.g., deep neural networks), which allowed us to develop an efficient approximated gradient descent method based on the Moreau envelope smoothing technique, inspired by recent advances in non-smooth DC optimization.
To increase the efficiency of large data processing, we used an efficient stochastic block coordinate update in our algorithm.
Our proposed algorithm can also be used to minimize the sum of ranked range loss, which also lacks efficient solvers.
We established a complexity of ˜ O
(1 /ϵ 6 ) for finding a nearly ϵ -critical solution.
Finally, we numerically demonstrated the effectiveness of our proposed algorithms in training both linear models and deep neural networks for partial AUC maximization and sum of ranked range loss minimization.
Introduction
The area under the receiver operating characteristic (ROC) curve (AUC) is one of the most widely used performance measures for classifiers in machine learning, especially when the data is imbalanced between the classes [7, 19].
Typically, the classifier produces a score for each data point.
Then a data point is classified as positive if its score is above a chosen threshold; otherwise, it is classified as negative.
Varying the threshold will change the true positive rate (TPR) and the false positive rate (FPR) of the classifier.
The ROC curve shows the TPR as a function of the FPR that corresponds to the same threshold.
Hence, maximizing the AUC of a classifier is essentially maximizing the classifier’s average TPR over all FPRs from zero to one.
However, for some applications, some FPR regions have no practical relevance.
So does the TPR over those regions.
For example, in clinical practice, a high FPR in diagnostic tests often results in a high monetary cost, so people may only need to maximize the TPR when the FPR is low [13, 34, 64].
Moreover, since two models with the same AUC can still have different ROCs, the AUC does not always reflect the true performance of a model that is needed in a particular production environment [8].
As a generalization of the AUC, the partial AUC (pAUC) only measures the area under the ROC curve that is restricted between two FPRs.
A probabilistic interpretation of the pAUC can be found in [13].
In contrast to the AUC, the pAUC represents the average TPR only over a relevant range of FPRs and provides a performance measure that is more aligned with the practical needs in some applications.
In literature, the existing algorithms for training a classifier by maximizing the pAUC include the boosting method [29] and the cutting plane algorithm
However, the former has no theoretical guarantee, and the latter applies only to linear models.
More importantly, both methods require processing all the data in each iteration and thus, become computationally inefficient for large datasets.
In this paper, we proposed an approximate gradient method for maximizing the pAUC that works for nonlinear models (e.g., deep neural networks) and only needs to process randomly sampled positive and negative data points of any size in each iteration.
In particular, we formulated the maximization of the pAUC as a non-smooth difference-of-convex (DC) program [30, 54].
Due to non-smoothness, most existing DC optimization algorithms cannot be applied to our formulation.
Motivated by [52], we approximate the two non-smooth convex components in the DC program by their Moreau envelopes and obtain a smooth approximation of the problem, which will be solved using the gradient descent method.
Since the gradient of the smooth problem cannot be calculated explicitly, we approximated the gradient by solving the two proximal-point subproblems defined by each convex component using the stochastic block coordinate descent (SBCD) method.
Our method, besides its low per-iteration cost, has a rigorous theoretical guarantee, unlike the existing methods.
In fact, we show that our method finds a nearly ϵ -critical point of the pAUC optimization problem in ˜ O
( ϵ − 6 ) iterations with only small samples of positive and negative data points processed per iteration.
1 This is the main contribution of this paper.
Note that, for non-convex non-smooth optimization, the existing stochastic methods [10, 11] find an nearly ϵ -critical point in O ( ϵ − 4 ) iterations under a weak convexity assumption.
Our method needs
O ( ϵ − 6 ) iterations because our problem is a DC problem with both convex components non-smooth which is much more challenging than a weakly non-convex minimization problem.
In addition, our iteration number matches the known best iteration complexity for non-smooth non-convex min-max optimization [33, 46] and non-smooth non-convex constrained optimization [35].
In addition to pAUC optimization, our method can be also used to minimize the sum of ranked range (SoRR) loss, which can be viewed as a special case of pAUC optimization.
Many machine learning models are trained by minimizing an objective function, which is defined as the sum of losses over all training samples [60].
Since the sum of losses weights all samples equally, it is insensitive to samples from minority groups.
Hence, the sum of top- k losses [17, 49] is often used as an alternative objective function because it provides the model with robustness to non-typical instances.
However, the sum of top- k losses can be very sensitive to outliers, especially when k is small.
To address this issue, [24] proposed the SoRR loss as a new learning objective, which is defined as the sum of a consecutive sequence of losses from any range after the losses are sorted.
Compared to the sum of all losses and the sum of top- k losses, the SoRR loss maintains a model’s robustness to a minority group but also reduces the model’s sensitivity to outliers.
See Fig.1 in [24] for an illustration of the benefit of using the SoRR loss over other ways of aggregating individual losses.
To minimize the SoRR loss, [24] applied a difference-of-convex algorithm (DCA) [3, 54], which linearizes the second convex component and solves the resulting subproblem using the stochastic subgradient method.
DCA has been well studied in literature; but when the both components are non-smooth, as in our problem, only asymptotic convergence results are available.
To establish the total number of iterations needed to find an ϵ -critical point in a non-asymptotic sense, most existing studies had to assume that at least one of the components is differentiable, which is not the case in this paper.
Using the approximate gradient method presented in this paper, one can find a nearly ϵ -critical point of the SoRR loss optimization problem in ˜ O
( ϵ − 6 ) iterations.
Related Works
The pAUC has been studied for decades [26, 37, 57, 65].
However, most studies focused on its estimation [13] and application as a performance measure, while only a few studies were devoted to numerical algorithms for optimizing the pAUC.
Efficient optimization methods have been developed for maximizing AUC and multiclass AUC by [69] and [66], but they cannot be applied to pAUC.
Besides the boosting method [29] and the cutting plane algorithm [41, 42, 43] mentioned in the previous section, [59, 67, 68, 73] developed surrogate optimization techniques that directly maximize a smooth approximation of the pAUC or the two-way pAUC [64].
However, their approaches can only be applied when the FPR starts from exactly zero.
On the contrary, our method allows the FPR to start from any value between zero and one.
[61] and [47] developed algorithms that use the pAUC as a criterion for creating a linear combination of multiple existing classifiers while we consider directly train a classifier using the pAUC.
DC optimization has been studied since the 1950s [2, 20].
We refer interested readers to [30, 45, 54, 55, 58], and the references therein.
The actively studied numerical methods for solving a DC program include DCA [3, 50, 54, 55], which is also known as the concave-convex procedure [32, 51, 72], the proximal DCA [5, 38, 40, 53], and the direct gradient methods [28].
However, when the two convex components are both non-smooth, the existing methods have only asymptotic convergence results except the method by [1], who considered a stopping criterion different from ours.
When at least one component is smooth, non-asymptotic convergence rates have been established with and without the
Kurdyka-Łojasiewicz (KL) condition [5, 6, 28, 50, 62].
The algorithms mentioned above are deterministic and require processing the entire dataset per iteration.
Stochastic algorithms that process only a small data sample per iteration have been studied [12, 21, 31, 36, 44].
However, they all assumed smoothness on at least one of the two convex components in the DC program.
The stochastic methods of [4, 56, 63] can be applied when both components are non-smooth but their methods require an unbiased stochastic estimation of the gradient and/or value of the two components, which is not available in the DC formulation of the pAUC maximization problem in this paper.
The technique most related to our work is the smoothing method based on the Moreau envelope [16, 18, 22, 23, 39, 52].
Our work is motivated by [39, 52], but the important difference is that they studied deterministic methods and assumed either that one function is smooth or that the proximal-point subproblems can be solved exactly, which we do not assume.
However, [39, 52] consider a more general problem and study the fundamental properties of the smoothed function such as its Lipschitz smoothness and how its stationary points correspond to those of the original problems.
We mainly focus on partial AUC optimization which has a special structure we can utilize when solving the proximal-point subproblems.
Additionally, [52] developed an algorithm when there were linear equality constraints, which we do not consider in this paper.
Preliminary
We consider a classical binary classification problem, where the goal is to build a predictive model that predicts a binary label y ∈ { 1 , − 1 } based on a feature vector x ∈ R p .
Let h w : R p
→ R be the predictive model parameterized by a vector w ∈ R d , which produces a score h w ( x ) for x .
Then x is classified as positive ( y = 1 ) if h w ( x ) is above a chosen threshold and classified as negative ( y = − 1 ), otherwise.
Let X + =
{ x + i } N + i and X − = { x − i } N − i be the sets of feature vectors of positive and negative training data, respectively.
The problem of learning h w through maximizing its empirical AUC on the training data can be formulated as
[Equation]
where 1 ( · ) is the indicator function which equals one if the inequality inside the parentheses holds and equals zero, otherwise.
According to the introduction, pAUC can be a better performance measure of h w than AUC.
Consider two FPRs α and β with 0 ≤ α < β ≤ 1 .
For simplicity of exposition, we assume N − α
and N − β are both integers.
Let m =
N − α and n = N − β .
The problem of maximizing the empirical pAUC with FPR between α and β can be formulated as
[Equation]
where [ j ] denotes the index of the j th largest coordinate in vector ( h w ( x − j ))
N − j =1 with ties broken arbitrarily.
Note that N + ( n − m ) in (2) is a normalizer that makes the objective value between zero and one.
Solving (2) is challenging due to discontinuity.
Let ℓ : R → R be a differential non-increasing loss function.
Problem (2) can be approximated by the loss minimization problem
[Equation]
To facilitate the discussion, we first introduce a few notations.
Given a vector S
=
( s i )
Ni =1 ∈ R N and an integer l with 0 ≤ l ≤ N , the sum of the top- l values in S is
[Equation]
where [ j ] denotes the index of the j th largest coordinate in S with ties broken arbitrarily.
For integers l 1 and l 2 with 0 ≤ l 1 < l 2 ≤ N , ϕ l 2 ( S ) − ϕ l 1 ( S ) is the sum from the ( l 1 +1)
th to the l 2 th (inclusive) largest coordinates of S , also called a sum of ranked range (SoRR).
In addition, we define vectors
[Equation]
for i = 1 , . . .
, N + , where s ij ( w ) :=
ℓ ( h w ( x + i ) − h w ( x − j )) for i
= 1 , . .
. , N + and j = 1 , . .
. , N − .
Since ℓ is non-increasing, the j th largest coordinate of S i ( w ) is ℓ ( h w ( x + i ) − h w ( x − [ j ] )) .
As a result, we have, for i = 1 , . . .
, N + ,
[Equation]
Hence, after dropping the normalizer, (3) can be equivalently written as
[Equation]
where
[Equation]
Next, we introduce an interesting special case of (5), namely, the problem of minimizing SoRR loss.
We still consider a supervised learning problem but the target y ∈ R does not need to be binary.
We want to predict y based on a feature vector x ∈ R p using h w ( x ) .
With a little abuse of notation, we measure the discrepancy between h w ( x ) and y by ℓ ( h w ( x ) , y ) , where ℓ : R 2 → R + is a loss function.
We consider learning the model’s parameter w from a training set D = { ( x j , y j ) } N j =1 , where x j ∈ R p and y j ∈ R for j = 1 , . . .
, N , by minimizing the SoRR loss.
More specifically, we define vector
[Equation]
Nj =1 , where s j ( w ) := ℓ ( h w ( x j ) , y j ) , j = 1 , . . .
, N .
Recall (4).
For any integers m and n with 0 ≤ m < n ≤ N , the problem of minimizing the SoRR loss with a range from m + 1 to n is formulated as min w { ϕ n ( S ( w ))
− ϕ m ( S ( w )) } , which is an instance of (5) with
[Equation]
If we view S i ( w ) and S ( w ) only as vector-value functions of w but ignore how they are formulated using data, (7) is a special case of (6) with N + = 1 and N − = N .
Nearly Critical Point and Moreau Envelope Smoothing
We first develop a stochastic algorithm for (5) with f l defined in (6).
To do so, we make the following assumptions, which are satisfied by many smooth h w ’s and ℓ ’s.
Assumption 1 (a) s ij ( w ) is smooth and there exists L ≥
such that 2 ∥∇ s ij ( w ) − ∇ s ij ( v ) ∥ ≤
L ∥ w − v ∥ for any w , v ∈ R d , i = 1 , . .
. , N + and j = 1 , . .
. , N − .
(b) There exists B ≥
such that ∥∇ s ij ( w ) ∥ ≤ B for any w ∈ R d , i = 1 , . .
. , N + and j = 1 , . .
. , N − .
(c) F ∗ > −∞ .
Given f : R d → R ∪ { + ∞} , the subdifferential of f is
[Equation]
where each element in ∂f ( w ) is called a subgradient of f at w .
We say f is ρ -weakly convex for some ρ ≥ 0 if f ( v ) ≥ f ( w ) + ⟨ ξ , v − w ⟩ − ρ 2 ∥ v − w ∥ 2 for any v and w and ξ ∈ ∂f ( w ) and say f is ρ -strongly convex for some ρ ≥ 0 if f ( v ) ≥ f ( w ) + ⟨ ξ , v − w ⟩ + ρ 2 ∥ v − w ∥ 2 for any v and w and ξ ∈ ∂f ( w ) .
It is known that, if f is ρ -weakly convex, then f ( w ) + 12 µ ∥ w ∥ 2 is a ( µ − 1 − ρ ) -strongly convex function when µ − 1 > ρ .
Under Assumption 1, ϕ l ( S i ( w )) is a composite of the closed convex function ϕ l and the smooth map
S i ( w ) .
According to Lemma 4.2 in [14], we have the following lemma.
Lemma 1 Under Assumption 1, f m ( w ) and f n ( w ) in (6) are ρ -weakly convex with ρ := N + N − L .
To solve (5) numerically, we need to overcome the following challenges.
(i) F ( w ) is non-convex even if each s ij ( w ) is convex.
In fact, F ( w ) is a DC function because, by Lemma 1, we can represent
F ( w ) as the difference of the convex functions f n ( w ) + 12 µ ∥ w ∥ 2 and f m ( w ) + 12 µ ∥ w ∥ 2 with µ − 1 > ρ .
(ii) F ( w ) is non-smooth due to ϕ l so that finding an approximate critical point (defined below) of F ( w ) is difficult.
(iii)
Computing the exact subgradient of f l ( w ) for l = m, n requires processing N + N − data pairs, which is computationally expensive for a large data set.
Because of challenges (i) and (ii), we have to consider a reasonable goal when solving (5).
We say w ∈ R d is a critical point of (5) if 0 ∈ ∂f n ( w ) − ∂f m ( w ) .
Given ϵ > 0 , we say w ∈ R d is an ϵ -critical point of (5) if there exists ξ ∈ ∂f n ( w ) − ∂f m ( w )
such that ∥ ξ ∥ ≤ ϵ .
A critical point can only be achieved asymptotically in general.
Within finitely many iterations, there also exists no algorithm that can find an ϵ -critical point unless at least one of f m and f n is smooth, e.g., [63].
Since f m and f n are both non-smooth, we have to consider a weaker but achievable target, which is a nearly ϵ -critical point defined below.
Definition 1 Given ϵ > 0 , we say w ∈ R d is a nearly ϵ -critical point of (5) if there exist ξ , w ′ , and w ′′ ∈ R d such that ξ ∈ ∂f n ( w ′ )
− ∂f m ( w ′′ ) and max {∥ ξ ∥ , ∥ w − w ′ ∥ , ∥ w − w ′′ ∥} ≤ ϵ .
Definition 1 is reduced to the ϵ -stationary point defined by [39, 52] when w equals w ′ or w ′′ .
However, obtaining their ϵ -stationary point requires exactly solving the proximal mapping of f m or f n while finding a nearly ϵ -critical point requires only solving the proximal mapping inexactly.
When w is generated by a stochastic algorithm, we also call w a nearly ϵ -critical point if it satisfies
Definition 1 with each ∥ · ∥ replaced by E ∥ · ∥ .
Motivated by [52] and [39], we approximate non-smooth F ( w ) by a smooth function using the
Moreau envelopes.
Given a proper, ρ -weakly convex and closed function f on R d , the Moreau envelope of f with the smoothing parameter µ ∈ (0 , ρ − 1 ) is defined as
[Equation]
and the proximal mapping of f is defined as
[Equation]
Note that the v µf ( w ) is unique because the minimization above is strongly convex.
Standard results show that f µ ( w ) is smooth with ∇ f µ ( w ) = µ − 1 ( w − v µf ( w )) and v µf ( w ) is (1 − µρ ) − 1 -Lipschitz continuous.
See Proposition 13.37 in [48] and Proposition 1 in [52].
Hence, using the Moreau envelope, we can construct a smooth approximation of (5) as follows
[Equation]
Function F µ has the following properties.
The first property is shown in [52].
We give the proof for the second in Appendix B.
Lemma 2 Suppose Assumption 1 holds and µ > ρ − 1 with ρ defined in Lemma 1.
The following claims hold
∇ F µ ( w ) = µ − 1 ( v µf m ( w ) − v µf n ( w )) and it is L µ -Lipschitz continuous with L µ = 2 µ − µ 2 ρ .
If ¯ v and w are two random vectors such that E ∥∇ F µ ( w ) ∥ 2 ≤ min { 1 , µ − 2 } ϵ 2 / 4 and E ∥ ¯ v − v µf l ( w ) ∥ 2 ≤ ϵ 2 / 4 for either l = m or l = n
then ¯ v is a nearly ϵ -critical points of (5) .
Since F µ is smooth, we can directly apply a first-order method for smooth non-convex optimization to (10).
To do so, we need to evaluate ∇ F µ ( w ) , which requires computing v µf m ( w ) and v µf n ( w ) , i.e., exactly solving (9) with f = f m and f = f n , respectively.
Computing the subgradients of f m and f n require processing N + N − data pairs which is costly.
Unfortunately, the standard approach of sampling over data pairs does not produce unbiased stochastic subgradients of f m and f n due to the composite structure ϕ l ( S i ( w )) .
In the next section, we will discuss a solution to overcome this challenge and approximate v µf m ( w ) and v µf n ( w ) , which leads to an efficient algorithm for (10).
Algorithm for pAUC Optimization
Consider (10) with f l defined in (6) for l = m and n .
To avoid of processing N + N − data points, one method is to introduce dual variables p
i =
( p ij )
N − j =1 for i = 1 , . . .
, N +
and formulate f l as
[Equation]
where P l = { p ∈ R N − | (cid:80) N − j =1 p j = l, p j ∈ [0 , 1] } .
Then (10) can be reformulated as a minmax problem and solved by a primal-dual stochastic gradient method (e.g. [46]).
However, the maximization in (11) involves N + N − decision variables and equality constraints, so the per-iteration cost is still O ( N + N − ) even after using stochastic gradients.
To further reduce the per-iteration cost, we take the dual form of the maximization in (11) (see
Lemma 4 in Appendix B) and formulate f l as
[Equation]
where λ = ( λ 1 , . .
, λ N + ) .
Hence, (9) with f = f l for l = m and n can be reformulated as
[Equation]
Note that g l ( v , λ ) is jointly convex in v and λ when µ − 1 > ρ = N + N − L (see Lemma 3 in
Appendix B).
Thanks to formulation (13), we can construct stochastic subgradient of g l and apply coordinate update to λ by sampling indexes
i ’s and j ’s, which significantly reduce the computational cost when N +
and N − are both large.
We present this standard stochastic block coordinate descent (SBCD) method for solving (13) in Algorithm 1 and present its convergence property as follows.
Proposition 1 Suppose Assumption 1 holds and µ − 1 > ρ = N + N − L , θ t = dist ( λ (0) , Λ ∗ ) √ ITN − and η t = ∥ v µfl ( w ) − w ∥ N + N − B √ T for any t in Algorithm 1.
It holds that
[Equation] [Equation]
where Λ ∗ = arg min λ g l ( v µf l ( w ) , λ ) .
Using Algorithm 1 to compute an approximation of v µf l ( w ) for l = m and n and thus, an approximation of ∇ F µ ( w ) , we can apply an approximate gradient descent (AGD) method to (10) and find a nearly ϵ -critical point of (5) according to Lemma 2.
We present the AGD method in Algorithm 2 and its convergence property as follows.
2: Set choose θ t
− 1 t =0 .
3: for
− 1 do 4:
Sample I t ⊂ { 1 , . .
+ with
| = and sample J
⊂ { 1 , . . .
, N − } with |J t | =
J .
Compute stochastic subgradient w.r.t.
v :
[Equation]
(cid:80)
[Equation]
Proximal subgradient update on v :
[Equation]
=
[Equation]
Compute stochastic subgradient w.r.t.
λ i for i ∈
I t :
[Equation] [Equation]
t
[Equation]
> λ ( t
[Equation]
(cid:17) for i ∈
I t 8:
Stochastic block subgradient update on λ
i for i ∈
I t : λ ( t +
[Equation]
i = λ ( t
[Equation]
i − θ t G ( t
[Equation]
i for i ∈
I t and λ ( t +1) i
= λ ( t ) i for i / ∈
I t .
(15) 9: end for 10: Output: (¯ v , ¯ λ ) =
1 T (cid:80) T − 1 t =
( v ( t ) , λ ( t ) ) .
[Equation] [Equation]
[Equation]
Theorem 1 Suppose Assumption 1 holds and Algorithm 1 is called in iteration k of Algorithm 2 with parameters µ − 1 > ρ
= N + N − L , θ t = dist (¯ λ ( k ) , Λ ∗ k )
√ IT k N − , η t = ∥ v µfl ( w ( k ) ) − w ( k ) ∥
N + N − B √ T k for any t , and
T k = max (cid:26) 144 N 2+ N 2 − D 2 l ( k + 1)
I ( µ − 1 − ρ ), 4 N 2+ N 2 − µ 2 l 2 B 2 ( k +
[Equation] [Equation]
µ − 1 − ρ ), 6 µl 2 B 2 ( k + 1) 2( µ − 1 − ρ )(cid:27) where Λ ∗ k = arg min λ g l ( v µf l ( w ( k ) ) , λ ) and
D l := max (cid:26) dist (¯ λ (0) l , Λ ∗ 0 ) ,  (cid:18)
[Equation]
Then ¯ v (¯ k ) n is a nearly ϵ -critical point of (5) with f l defined in (6) with K no more than
K
[Equation]
According to Theorem 1, to find a nearly ϵ -critical point of (5), we need K =
˜
O
( ϵ − 2 ) iterations in
Algorithm 2 and (cid:80) K − 1 k =
T k
= O
( K 3 )
=
˜ O
( ϵ − 6 ) iterations of Algorithm 1 in total across all calls.
Remark 1 (Challenges in proving Theorem 1) Suppose we can set T k in lines 3 and 4 of Algorithm 2 appropriately such that the approximation errors E ∥
¯ v ( k ) m − v µf m ( w ( k ) )
∥ 2 and E ∥ ¯ v ( k ) n −
[Figure] [Figure]
T r a i n i ng
Lo ss
[Figure] [Figure]
[Figure] [Figure]
epoch 0.
[Figure] [Figure]
[Figure] [Figure]
[Figure]
p AUC on T r a i n i ng se t
[Figure]
v µf n ( w ( k ) )
∥ 2 are both O (1 /k ) .
We can then prove that Algorithm 2 finds a nearly ϵ -critical point within K =
˜ O
( ϵ − 2 ) iterations and the total complexity is (cid:80) K −
1 k =
T k .
This is just a standard idea.
However, by Proposition 1, such a T k must be Θ( k 2 ( dist 2 (¯ λ ( k ) , Λ ∗ k ) + ∥ v µf l ( w ( k ) ) − w ( k ) ∥ 2 )) where dist 2 (¯ λ ( k ) , Λ ∗ k ) and ∥ v µf l ( w ( k ) )
− w ( k ) ∥ 2 also change with k .
Then it is not clear what the order of T k is.
By a novel proving technique based on the (linear) error-bound condition of g l ( w , λ ) with respect to λ , we prove that both dist 2 (¯ λ ( k ) , Λ ∗ k ) and ∥ v µf l ( w ( k ) )
− w ( k ) ∥ 2 are O (1) (see (27) and (30) in Appendix D) which ensures that T k = Θ( k 2 ) and thus the total complexity is (cid:80) K −k =
T k
= O
( K 3 )
=
˜ O
( ϵ − 6 ) .
Remark 2 (Analysis of sensitivity of the algorithm to µ ) For the interesting case where ρ ≥ 1 , we have µ < 1 /ρ < 1 .
In this case, we can derive that the order of dependency on µ is O ( 1 ϵ 6 µ 6 ) and the optimal choice of µ is thus Θ( ρ − 1 ) , e.g., µ = 12 ρ , which leads to a complexity of O ( ρ 6 /ϵ 6 ) .
We present the convergence curves and the test performance of our method when applied to training a linear model with µ of different values in Appendix E.7.
The technique in the previous sections can be directly applied to minimize the SoRR loss, which is formulated as (5) but with f l defined in (7).
Due to the limit of space, we present the algorithm for minimizing the SoRR loss and its convergence result in Appendix A.
Numerical Experiments
In this section, we demonstrate the effectiveness of our algorithm AGD-SBCD for pAUC maximization and SoRR loss minimization problems (see Appendix E.1 for details).
All experiments are conducted in Python and Matlab on a computer with the CPU 2GHz Quad-Core Intel Core i5 and the
GPU NVIDIA GeForce RTX 2080 Ti.
All datasets we used are publicly available and contain no personally identifiable information and offensive contents.
6.1 Partial AUC Maximization
For maximizing pAUC, we focus on large-scale imbalanced medical dataset CheXpert [25], which is licensed under CC-BY-SA and has 224,316 images.
We construct five binary classification tasks with the logistic loss ℓ ( z )
=
log(1 + exp( − z )) for predicting five popular diseases, Cardiomegaly (D1),
Edema (D2), Consolidation (D3), Atelectasis (D4), and P. Effusion (D5).
For comparison of training convergence, we consider different methods for optimizing the partial
AUC.
We compare with three baselines, DCA [24] (see Appendix E.4 for details), proximal DCA [62] (see Appendix E.5 for details) and SVM pAUC -tight [42].
Since DCA, proximal DCA and SVM pAUCtight cannot be applied to deep neural networks, we focus on linear model and use a pre-trained deep neural network to extract a fixed dimensional feature vectors of 1024 .
The deep neural network was trained by optimizing the cross-entropy loss following the same setting as in [70].
For three baselines and our algorithm, the process to tune the hyper-parameters is explained in
Appendix E.2.
In Figure 1 and Figure 3 in Appendix E.3, we show how the training loss (the objective value of (3)) and normalized partial AUC on the training data change with the number of epochs.
[Table]
We observe that for all of these five diseases, our algorithm converges much faster than DCA and proximal DCA and we get a better partial AUC than DCA and proximal DCA.
The comparison between our AGD-SBCD and SVM pAUC -tight on training data are shown in Table 1.
As shown from the second to the fifth column of Table 1, our algorithm needs only a few seconds to exceed the pAUCs that SVM pAUC -tight takes more than one minute to return.
As shown from sixth to eighth column, our algorithm eventually improves the pAUC by at least 12% compared with SVM pAUC -tight.
DCA and proximal DCA are not included in the tables because it computes deterministic subgradients, which leads to a runtime significantly longer than the other two methods.
We plot the convergence curves of training pAUC over GPU time for DCA and our algorithm in
Figure 7 in Appendix E.8.
To compare the testing performances, we consider the deep neural networks besides the linear model.
For linear model, we still compare with DCA and SVM pAUC -tight.
For deep neural networks, we compare with the naive mini-batch based method (MB) [27] and methods based on different optimization objectives, including the cross-entropy loss (CE) and the AUC min-max margin loss (AUC-M) [71].
We learn the model DenseNet121 from scratch with the CheXpert training data split in train/
val=9:1 and the CheXpert validation dataset as the testing set, which has 234 samples.
The range of FPRs in pAUC is [0.05, 0.5].
For optimizing CE, we use the standard Adam optimizer.
For optimizing AUC-M, we use the PESG optimizer in [71].
We run each method 10 epochs and the learning rate ( c in AGD-SBCD) of all methods is tuned from { 10 − 5 ∼ 10 0 } .
The minibatch size is 32.
For AGD-SBCD, T k is set to 50( k + 1) 2 , µ is set to 10 3 N + N − and γ is tuned from { 0 . 1 , 1 , 2 } × 10 3 / ( N + N − ) .
For MB, the learning rate decays in the same way as in [27].
For
CE and AUC-M, the learning rate decays 10-fold after every 5 epochs.
For AUC-M, we tune the hyperparameter γ in {100, 500, 1000}.
For each method, the validation set is used to tune the hyperparameters and select the best model across all iterations.
The results of the pAUCs on the testing set are reported in Table 2, which shows that our algorithm performs the best for all diseases.
The complete ROC curves on the testing set are shown in Appendix E.
[Table] [Table]
For deep neural networks
[Table]
we also learn the model ResNet-20 from scratch with the CIFAR-10-LT and the Tiny-ImageNet-200-LT datasets, which are constructed similarly as in [67].
Details about these two datasets are summarized in Appendix E.6.
The range of FPRs in pAUC is [0.05, 0.5].
The process of tuning hyperparameters is the same as that for CheXpert.
The results of the pAUCs on the testing set are reported in Table 3, which shows that our algorithm performs the best for these two long-tailed datasets.
[Table]
Conclusion
Most existing methods for optimizing pAUC are deterministic and only have an asymptotic convergence property.
We formulate pAUC optimization as a non-smooth DC program and develop a stochastic subgradient method based on the Moreau envelope smoothing technique.
We show that our method finds a nearly ϵ -critical point in ˜ O
( ϵ − 6 ) iterations and demonstrate its performance numerically.
A limitation of this paper is the smoothness assumption on s ij ( w ) , which does not hold for some models, e.g., neural networks using ReLU activation functions.
It is a future work to extend our results for non-smooth models.
Acknowledgements
This work was jointly supported by the University of Iowa Jumpstarting Tomorrow Program and NSF award 2147253.
T. Yang was also supported by NSF awards 2110545 and 1844403, and Amazon research award.
We thank Zhishuai Guo, Zhuoning Yuan and Qi Qi for discussing about processing the image dataset.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Algorithm 1 and 2 and Section 6.
(b) Did you describe the limitations of your work?
[Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[No] We are not aware of any negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Section 6.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 6.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 6.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 6. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite them as [9, 15, 25] (b) Did you mention the license of the assets?
[Yes] See Section 6.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A]
We only use publicly available data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See Section 6.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A Algorithm for Sum of Range Optimization
The technique in the previous sections can be directly applied to minimize the SoRR loss, which is formulated as (5) but with f l defined in (7).
Since (7) is a special case of (6) with N + = 1 and
N − =
N , we can again formulate subproblem (9) with f = f l as (13) with λ =
λ being a scalar.
Since λ is a scalar, when solving (13), we no longer use block coordinate update but only need to sample over indexes j = 1 , . .
, N to construct stochastic subgradients.
We present the stochastic subgradient (SGD) method for (13) in Algorithm 3.
Next, we apply Algorithm 2 with SBCD in lines 3 and 4 replaced by SGD.
The convergence result in this case is directly from Theorem 1.
[Equation]
update
[Equation] [Equation]
λ = l − N J (cid:88)
j ∈J t 1 (cid:16) s j ( v ( t ) )
> λ ( t ) (cid:17) 8:
Stochastic subgradient update on λ : λ ( t +1) =
λ ( t ) − η t G ( t ) λ 9: end for 10: Output: (¯ v , ¯ λ )
=
1 T (cid:80) T − 1 t =
( v ( t ) , λ ( t ) ) .
Corollary 1 Suppose Assumption 1 holds with N + = 1 , N − =
N and s ij ( w ) =
s j ( w ) and SBCD in Algorithm 2 are replaced by SGD (Algorithm 3).
Suppose θ t , η t , and T k are set the same as in
Theorem 1 when Algorithm 3 is called in iteration k of Algorithm 2.
Then ¯ v (¯ k ) n is an nearly ϵ -critical point of (5) with f l defined in (7) with K no more than (17) .
B Proofs of Lemmas
Proof.
[of Lemma 2] We will only prove the second conclusion in Lemma 2 since the first conclusion has been shown in Proposition 1 in [52].
Suppose E ∥∇ F µ ( w ) ∥ 2 ≤ min { 1 , µ − 2 } ϵ 2 / 4 .
By the first conclusion in Lemma 2, we must have µ − 2 E ∥ v µf m ( w ) − v µf n ( w ) ∥ 2 ≤ min { 1 , µ − 2 } ϵ 2 / 4 .
By the optimality conditions satisfied by v µf m ( w ) and v µf n ( w ) , there exist ξ m ∈
∂f m ( v µf m ( w )) and ξ n ∈ ∂f n ( v µf n ( w )) such that ξ m + µ − 1 ( v µf m ( w ) − w ) =
= ξ n + µ − 1 ( v µf n ( w ) − w ) , which implies ξ = ξ n − ξ m = µ − 1 ( v µf m ( w ) − v µf n ( w )) ∈
∂f n ( v µf n ( w )) − ∂f m ( v µf m ( w )) and
E ∥ ξ ∥ ≤ (cid:112) E ∥ ξ ∥ 2 ≤ ϵ/ 2 .
Suppose E ∥ ¯ v − v µf n ( w ) ∥ 2 ≤ ϵ 2 / 4 .
We have E ∥ ¯ v − v µf n ( w ) ∥ ≤ ϵ/and E
∥
¯ v − v µf m ( w ) ∥ ≤ E ∥ ¯ v − v µf n ( w ) ∥ + E ∥ v µf n ( w ) − v µf m ( w ) ∥ ≤ ϵ .
Hence, ¯ v satisfies
Definition 1 with w ′ = v µf n ( w ) and w ′′ = v µf m ( w ) .
Suppose E ∥ ¯ v − v µf m ( w ) ∥ 2 ≤ ϵ 2 / 4 .
The conclusion can be also proved similarly.
□
We first present the following lemma which is similar to Lemma 4.2 in [14].
Lemma 3 Suppose Assumption 1 holds.
For any v , λ , v ′ , λ ′ , and ( ξ v , ξ λ ) ∈ ∂g
l ( v ′ , λ ′ ) , we have
[Equation]
where ρ = N + N − L .
Moreover, g l ( v , λ ) + 12 µ ∥ v − w ∥ 2 is jointly convex in λ and v for any w and any µ − 1 > ρ .
Proof.
By Assumption 1, we have
[Equation] [Equation]
Let ξ ij ∈ ∂
[ s ij ( v ′ ) − λ ′
i ]
+ .
We have g l ( v , λ ) =
l 1 ⊤ λ +
N + (cid:88) i =
N − (cid:88)
j
=
[ s ij ( v ) − λ
i ] + ≥ l 1 ⊤ λ ′ + l 1 ⊤ ( λ − λ ′ )
+
N + (cid:88) i =
N − (cid:88)
j
=
[ s ij ( v ′ ) − λ ′
i ] + +
N + (cid:88) i =
N − (cid:88)
j = ξ ij ( s ij ( v ) − s ij ( v ′ )
− λ i + λ ′ i ) ≥ g l ( v ′ , λ ′ )
+
ξ ⊤ λ ( λ − λ ′ )
+
N + (cid:88) i =
N − (cid:88)
j = ξ ij ∇ s ij ( v ′ )
⊤ ( v − v ′ )
−
N + (cid:88) i =
N − (cid:88)
j = ξ ij L 2 ∥
v − v ′ ∥≥ g l ( v ′
[Equation]
where the first inequality is by the convexity of [ · ] + , the second inequality is from (18) and the last inequality is by the definitions of ( ξ v , ξ λ ) and the fact that ξ ij ∈ [0 , 1] .
Combining the inequality from the first conclusion with the equality 12 µ ∥ v − w ∥ 2 = 12 µ ∥ v − v ′
∥ 2 + 1 µ ( v − v ′ )
⊤ ( v ′ − w ) + 12 µ ∥ v ′ − w ∥ 2 and using the fact that µ − 1 > ρ , we can obtain
[Equation]
which proves the second conclusion.
□
Lemma
The dual problem of the maximization problem in (11) is the minimization problem in (12) .
Proof.
For i = 1 , . . .
, N + , we introduce a Lagrangian multiplier
λ
i for the constraint (cid:80) N − j =1 p j = l in (11).
Let [ z ] − = min { z, 0 } which equals − [ − z ] + .
Then, for each i , we have
[Equation]
 
N
[Equation]
j
[Equation]
ij
[Equation]
ij (
[Equation]
 
[Equation]
  −
[Equation]
(cid:88)
j
[Equation]
w
[Equation]
λ
[Equation]
j
[Equation]
p
[Equation]
  =
[Equation]
  − lλ
[Equation]
+
N − (cid:88)
j = p ij [ λ
[Equation]
The conclusion is thus proved by summing up the equality above for i = 1 , . . .
, N + .
□
C Proof of Proposition
Let v ∗ = v µf l ( w ) be the unique optimal solution of (8) and let s i [ j ] ( v ∗ ) be the j th largest coordinate of S i ( v ∗ ) .
It is easy to show that the set of optimal solutions of (13) is { v ∗ } × Λ ∗ where
[Equation]
Given any λ ∈ R N + , we denote its projection onto Λ ∗ as Proj Λ ∗ ( λ ) .
By the structure of Λ ∗ , the i th coordinate of Proj Λ ∗ ( λ ) is just the projection of λ i onto Λ ∗ i , which we denote by Proj Λ ∗
i ( λ i ) .
Moreover, we denote the distance from λ to Λ ∗ as dist ( λ , Λ ∗ ) and it satisfies
[Equation]
With these definitions, we can present the following lemma.
Lemma 5 Suppose Assumption 1 holds and µ − 1 > ρ .
For any w , v , λ and v ∗ = v µf l ( w ) , we have
N
[Equation]
B ∥ v
[Equation]
v
[Equation]
v
[Equation]
v ∗ , Proj Λ ∗
[Equation]
λ )) ≥
[Equation]
Λ
[Equation] [Equation]
Proof.
It is easy to observe that g l ( v , λ ) :
= (cid:80) N
+ i =1 g li ( v , λ i ) , where g li ( v , λ i ) :
= lλ i +
N − (cid:88)
[Equation]
[ s
[Equation]
and Λ ∗ i =
arg min λ
i g li ( v ∗ , λ i ) .
Since g li ( v ∗ , λ i ) is a piecewise linear in λ i with an outward slope of at least one at either end of the interval Λ ∗ i , we must have
[Equation]
which implies
[Equation]
Moreover, by Assumption 1, g li ( v , λ i ) is B -Lipschitz continuous in v so g l ( v , λ ) is N + B -Lipschitz continuous in v .
We then have N + B ∥ v − v ∗ ∥ + g l ( v , λ ) ≥ g l ( v ∗ , λ ) , which implies the conclusion together with the previous inequality.
□
We present the proof of Proposition 1 below.
Proof.
[of Proposition 1] Let us denote I [ t ] = {I 0 , . .
, I t } and J [ t ] =
{J 0 , . . .
, J t } .
Let E t be the expectation conditioning on I [ t − 1] and J [ t − 1] .
By Assumption 1 and the definitions of G ( t ) v and G ( t ) λ i in Algorithm 1, we have ∥ G ( t ) v ∥ ≤ N + N − B and | G ( t )
λ i | ≤ N − for t = 0 , . . .
, T − 1 and i = 1 , . . .
, N + .
By the optimality condition satisfied by v ( t +1) and (cid:16) 1 µ + 1 η t (cid:17) -strong convexity of the objective function in (14), we have
[Equation]
where the last inequality is by Young’s inequality.
Since v ∗ − v ( t ) is deterministic conditioning on
I [ t − 1] and J [ t − 1] , applying (21) and taking expectation E t on the both sides of the inequality above yield
[Equation]
By the updating equation (15) for λ ( t +1)
[Equation]
we have
[Equation]
Since λ ( t ) i − Proj Λ ∗
i ( λ ( t ) i ) is deterministic conditioning on I [ t − 1] and J [ t − 1]
[Equation]
applying (21) and taking expectation E t on the both sides of the inequality above yield
[Equation]
Multiplying both sides of (23) by N + 2 Iθ t and adding it with (22)
[Equation]
we have
[Equation] [Equation]
where the second inequality is because ( E t G ( t ) v , E t G ( t ) λ 1 , . .
, E t G ( t ) λ N + ) ∈ ∂g
l ( v ( t ) , λ ( t ) ) , which allows us to apply Lemma 3 with ( v , λ ) =
( v ∗ , Proj Λ ∗ ( λ ( t ) )) and ( v ′ , λ ′ )
= ( v ( t ) , λ ( t ) ) .
Notice that θ t and η t do not change with t that ρ < µ − 1 .
Summing up (24) for t = 0 , . .
, T − 1 , taking full expectation, and organizing terms give us
[Equation]
Because f l (¯ v ) + 12 µ ∥ ¯ v − w ∥ 2 is ( µ − 1 − ρ ) -strongly convex and the facts that f l (¯ v ) ≤ g l (¯ v , ¯ λ ) and that f l ( v ∗ ) =
g l ( v ∗ , λ ∗ ) for any optimal λ ∗ , we have that
[Equation]
where the last inequality is because g l ( v , λ ) + 12 µ ∥ v − w ∥ 2 is jointly convex in v and λ .
Recall that v (0) =
w .
Applying (25) to the left-hand side of the inequality above, we obtain
[Equation]
IT
[Equation]
The conclusion is then proved given that v µf l ( w )
= v ∗ .
□
D Proof of Theorem
We first present a technical lemma.
Lemma
Given two intervals [ a, b ] and [ a ′ , b ′ ] with max {| a − a ′ | , | b − b ′ |} ≤ δ .
We have dist ( z, [ a, b ]) ≤ dist ( z, [ a ′ , b ′ ])
+ δ, ∀ z ∈ R .
Proof.
We will prove the result in three cases, z < a ′ , z > b ′ and a ′ ≤ z ≤ b ′ .
Suppose z < a ′ so that dist ( z, [ a ′ , b ′ ]) =
| z − a ′ | .
We have dist ( z, [ a, b ]) ≤ |
z − a | ≤ | z − a ′ | + | a − a ′ | ≤ dist ( z, [ a ′ , b ′ ])+ δ .
The result when z > b ′ can be proved similarly.
Suppose a ′ ≤ z ≤ b ′ so that dist ( z, [ a ′ , b ′ ]) = 0 .
Note that z
= z − a ′ b ′ − a ′ · b ′ + b ′ − z b ′ − a ′ · a ′ .
We define z ′ =
z − a ′ b ′ − a ′ · b + b ′ − z b ′ − a ′ · a ∈ [ a, b ] .
Then dist ( z, [ a, b ]) ≤ | z − z ′
| ≤ z − a ′ b ′ − a ′ · | b − b ′ | + b ′ − z b ′ − a ′ · | a − a ′ | ≤ dist ( z, [ a ′ , b ′ ]) + δ .
□
Under Assumption 1, we have ∥∇ s ij ( v ) ∥
≤ B for any v so that ∥ ξ ∥ ≤ lB for any ξ ∈ ∂f l ( v ) and any v .
By the definition of v µf l ( w ) , we have µ − 1 ( w − v µf l ( w ))
∈
∂f l ( v µf l ( w )) , which implies
[Equation]
any w .
[Equation]
We then provide the following proposition as an additional conclusion from the proof of Proposition 1.
Proposition 2 Suppose Assumption 1 holds.
Algorithm 1 guarantees that
[Equation]
Proof.
By (25), the last inequality in (26), and Lemma 5, we have
[Equation]
which implies the conclusion by reorganizing terms.
□
Next we are ready to present the proof of of Theorem 1.
Proof.
[ of Theorem 1 ] Let ϵ k = 1 √ k +1 for k = 1 , . . . .
In the k th iteration of Algorithm 2, Algorithmis applied to the subproblem
[Equation]
with initial solution ( w ( k ) , ¯ λ ( k ) l ) and runs for T k iterations.
Let Λ ∗ k be the set of optimal λ for (28).
We will prove by induction the following two inequalities for k = 0 , 1 , . . . .
[Equation]
where D l is defined in (16) for l = m and n .
Applying Proposition 1 to (28) and using (27), we have, for k = 0 , 1 , . .
[Equation]
Moreover, applying Proposition 2 to (28) and using (27), we have
N + (cid:88)
[Equation]
Suppose k = 0 .
By (31) and the choice of T 0 , we have E ∥ ¯ v (0) l
[Equation]
∥
[Equation]
ϵ
[Equation]
/ 4 , so (29) holds for k = 0 .
In addition, (30) holds trivially for k = 0 .
Next, we assume (29) and (30) both hold for k and prove they also hold for k + 1 .
Since (29) and (30) hold for k , by (32) and the choice of T k , we have
N + (cid:88)
i
[Equation]
≤
[Equation]
From the updating equation w ( k +1) =
w ( k ) − γµ − 1 (¯ v ( k ) m
− ¯ v ( k ) n ) in Algorithm 2, we know that
[Equation]
where the second inequality is by triangle inequality and the last inequality is by (27) and the fact that (29) holds for k .
Let ¯ λ ( k +1) l,i denote the i th coordinate of ¯ λ ( k +1) l for i = 1 , . . .
N + and l = m and n .
Recall that s i
[ j ] ( v ) be the j th largest coordinate of S
i ( v ) .
By Assumption 1 and elementary argument, we can show that s i
[ j ] ( v ) is B -Lipschitz continuous for any i and j .
Since v µf l ( w ) is 1 1 − µρ -Lipschitz continuous, we have
[Equation]
for j = l and j = l + 1 , where the last inequality is by (34).
According to (19), (20), (35), and
Lemma 6, we can prove that, for i = 1 , . . .
, N + ,
E
[Equation]
+
[Equation]
Combining this inequality with (33) yields (30) for case k + 1 .
Stating (31) for case k + 1 gives
[Equation]
By (36), (30) for case k +1 , and the choice of T k
+1 , we have E ∥ ¯ v ( k +1)
l
[Equation]
which means (29) holds for case k + 1 .
By induction, we have proved that (29) and (30) holds for k = 0 , 1 , . . . .
Because ∇ F µ ( w ) = µ − 1 ( v µf m ( w ) − v µf n ( w )) is L µ -Lipschitz continuous, we have
[Equation]
Applying Young’s inequality to the first term on the right-hand side of the last inequality above gives
[Equation]
where the second inequality is because of (30) for l = m and n and the last because of γ ≤ 1 L µ .
Summing the above inequality over k = 0 , · · ·
[Equation]
K − 1 , we have
[Equation]
where the second inequality is because F ∗ ≤ F ( v µf n ( w ( K ) ))
≤ F µ ( w ( K ) ) and F µ ( w (0) ) ≤
F ( v µf m ( w (0) ))
(see Lemma 1 in [52]).
Let ¯ k be randomly sampled from { 0 , . .
, K − 1 } , then it holds
E
[Equation]
by the definition of K .
On the other hands, by (29), we have for l = m and n that
[Equation]
where the last inequality is because of the value of K .
Hence, by the second claim in Lemma(with w = w (¯ k ) and ¯ v = ¯ v (¯ k ) n ), ¯ v (¯ k ) n is an nearly ϵ -critical point of (5) with f l defined in (6).
This completes the proof.
□
E Additional Materials for Numerical Experiments
In this section, we present some additional details of our numerical experiments in Section 6 which we are not able to show due to the limit of space.
E.1 Details of SoRR Loss Minimization Experiment
For SoRR loss minimization, we compare with the baseline DCA.
We focus on learning a linear model h w ( x ) =
w T x with four benchmark datasets from the UCI [15] data repository preprocessed by Libsvm [9]: a9a, w8a, ijcnn1 and covtype.
The statistics of these datasets are summarized in
Table 4.We use logistic loss l ( h w ( x ) , y )
= − y log
( h w ( x )) − (1 − y )
log(1 − h w ( x )) where label y ∈ { 0 , 1 } .
Due to the limit space, we present the process of tuning hyperparameters and the convergence curves (Figure 2).
From the curves, we notice that our algorithm reduce SoRR loss faster than DCAs for all of these four datasets.
In this section, we first present some summary statistics on the datasets.
For AGD-SBCD, we fix the J = 100 , m = 10 3 and n = 2 × 10 4 .
In the spirit of Corollary 1, within
Algorithm 3 called in the k th main iteration, η t and θ t are both set to c ( k +1) N for any t with c tuned in
[Table]
the range of { 0 . 1 , 2 , 5 , 10 , 1 } and T k is set to C ( k +1) 2 with C selected from { 30 , 50 , 100 } .
Parameter µ is chosen from { 2 × 10 2 , 10 3 } /N and γ is tuned from { 2 × 10 2 , 5 × 10 2 , 8 × 10 2 , 4 × 10 3 , 2 × 10 4 } /N .
According to the experiments in [24], when solving (37), we first use the same step size and the same number of iterations for all k ’s.
We choose the step size from { 0 . 01 , 0 . 1 , 0 . 5 , 1 } and the iteration number from { 2000 , 3000 } .
However, we find that the performance of DCA improves if we let the step size and the number of iterations vary with k .
Hence, we apply the same process as in AGD-SBCD to select θ t , η t , and T in Algorithm 4 in the k th main iteration of DCA.
We report the performances of DCA under both settings (named DCA.Constant.lr and DCA.Changing.lr, respectively).
The changes of the SoRR loss with the number of epochs are shown in Figure 2.
From the curves, we notice that our algorithm reduce SoRR loss faster than DCAs for all of these four datasets.
[Figure]
100 # of epoch 0.
T r a i n i ng
Lo ss ijcnn1 AGD-SGDDCA.Constant.lrDCA.Changing.lr 5 10 15 20 25 30 # of epoch  3
T r a i n i ng
Lo ss covtype AGD-SGDDCA.Constant.lrDCA.Changing.lr  15 0.
E.2 Process of Tuning Hyperparameters
For AGD-SBCD, we fix the I
=
J = 100 , m = 10 4 and n = 10 5 .
In the spirit of Theorem 1, within Algorithm 3 called in the k th main iteration, η t is set to c ( k +1) N + N − , θ t is set to c ( k +1)
N − for any t with c tuned in { 0 . 1 , 1 , 5 , 10 , 20 , 30 } and T k is set to 50( k + 1) 2 .
Parameters µ is chosen from { 10 2 , 10 3 } / ( N + N − ) and γ is tuned from { 2 × 10 3 , 5 × 10 3 } / ( N + N − ) .
For DCA, we only implement the setting of DCA.Changing.lr (see Appendix E.1 for descriptions).
In Algorithm 4 called in the k th main iteration, η t and θ t are selected following the same process as in AGD-SBCD and T is set to C ( k + 1) 2 with C chosen from { 100 , 200 , 500 } .
For the SVM pAUC -tight method, we use their MATLAB code in [42] and tune hyper-parameter C from { 10 − 3 , 10 − 2 , 10 − 1 , 10 0 } .
E.3 Additional Plots of Partial AUC Maximization Experiment
The results for partial AUC maximization of diseases D3 ∼ D5 are shown in Figure 3.
We plot the ROC curves of three algorithms with linear model on the CheXpert testing set in Figure 4.
The range of false positive rate for the pAUC is set as [0 . 05 , 0 . 5] .
We plot the convergence curves of patial AUC on the CheXpert testing set of our algorithm AGDSBCD and baseline MB in Figure 5.
E.4 DCA and Algorithm for its Subproblem
Although DCA is originally only studied for SoRR loss minimization in [24], it can also be applied to pAUC maximization.
Hence, we only describe DCA for pAUC maximization which cover SoRR loss minimization as a special case.
At the k th iteration of DCA, it computes a deterministic subgradient of f m ( w ) =
(cid:80) N
+ i =1 ϕ m ( S i ( w ( k ) )) at iterate w ( k ) , denoted by ξ ( k ) .
Then DCA updates w ( k ) by
[Figure]
t
[Figure]
D4 and D5.
[Figure]
approximately solving the following subproblem using a SBCD method similar to Algorithm 1 by sampling indexes i and j (see Algorithm 4 for details).
[Equation]
Algorithm 4 is used to solve the subproblem (37) in the k th main iteration of DCA.
It is similar to the
SBCD method in Algorithm 1.
E.5 Details about Proximal DCA
At the k th iteration of proximal DCA, it computes a deterministic subgradient of f m ( w ) =
(cid:80) N
+ i =1 ϕ m ( S i ( w ( k ) )) at iterate w ( k ) , denoted by ξ ( k ) .
Then proximal DCA updates w ( k ) by approximately solving the subproblem
[Equation]
min
[Equation]
[Figure]
} with |J t | = J .
Compute stochastic subgradient w.r.t.
w :
G ( t ) w = N + N − IJ (cid:88)
i ∈I t (cid:88)
j ∈J t ∇ s ij ( w ( t ) ) 1 (cid:16) s ij ( w ( t ) )
> λ ( t ) i (cid:17) − ξ ( k ) 7:
Stochastic subgradient update on w : w ( t +1) =
w ( t ) − η t G ( t ) w 8:
Compute stochastic subgradient w.r.t.
λ i for i ∈
I t :
G ( t ) λ i = n − N − J (cid:88)
j ∈J t 1 (cid:16) s ij ( w ( t ) )
> λ ( t ) i (cid:17) for i ∈
I t 9:
Stochastic block subgradient update on λ
i for i ∈
I t : λ ( t +1) i
= (cid:40)
λ ( t ) i − θ t G ( t )
λ
i i ∈
I t , λ ( t ) i
i / ∈
I t .
10: end for 11: Output: ( ¯ w , ¯ λ ) =
( w ( T ) , λ ( T ) ) .
using a SBCD method similar to Algorithm 1 by sampling indexes i and j .
In proximal DCA, L is tuned from { 10 − 5 ∼ 10 0 } and other hyper-parameters are tuned from the same range as DCA.
E.6 Details about CIFAR-10-LT and Tiny-Imagenet-200-LT Datasets
Binary CIFAR-10-LT Dataset .
To construct a binary classification, we set labels of category ’cats’ to 1 and labels of other categories to 0.
We split the training data in train/val = 9:1 and use the validation dataset as the testing set.
More details are provided in Table 5.
Binary Tiny-Imagenet-200-LT Dataset .
To construct a binary classification, we set labels of category ’birds’ to 1 and labels of other categories to 0.
We split the training data in train/val = 9: and use the validation dataset as the testing set.
More details are provided in Table 5.
[Table]
E.7 Convergence Curves and Testing Performance of AGD-SBCD on Different µ
[Table]
E.8 Convergence Curves of Training pAUC over GPU Time
[Figure] [Figure]