On the consistent estimation of optimal Receiver Operating Characteristic (ROC) curve
Abstract
Under a standard binary classiﬁcation setting with possible model misspeciﬁcation, we study the problem of estimating general Receiver Operating Characteristic (ROC) curve, which is an arbitrary set of false positive rate (FPR) and true positive rate (TPR) pairs.
We formally introduce the notion of optimal ROC curve over a general model space.
It is argued that any ROC curve estimation methods implemented over the given model space should target the optimal ROC curve over that space.
Three popular ROC curve estimation methods are then analyzed at the population level (i.e., when there are inﬁnite number of samples) under both correct and incorrect model speciﬁcation.
Based on our analysis, they are all consistent when the surrogate loss function satisﬁes certain conditions and the given model space includes all measurable classiﬁers.
Interestingly, some of these conditions are similar to those that are required to ensure classiﬁcation consistency.
When the model space is incorrectly speciﬁed, however, we show that only one method leads to consistent estimation of the ROC curve over the chosen model space.
We present some numerical results to demonstrate the effects of model misspeciﬁcation on the performance of various methods in terms of their ROC curve estimates.
Introduction
One common goal in standard binary classiﬁcation task is to ﬁnd a classiﬁer with minimum misclassiﬁcation error, which is the weighted average of the false negative rate (FNR) and false positive rate (FPR).
This implicitly assumes that the two types of errors are equally important.
In many real applications, however, these two types of errors may need to be treated unequally by assigning asymmetric losses to them.
For example, in spam email detection, classifying non-spam as spam may incur a higher cost than classifying spam into non-spam.
Existing methods to handle this problem can be roughly categorized into three groups.
Cost-sensitive learning [Elkan, 2001], which generalizes the empirical risk minimization framework [Vapnik, 1991], assigns unequal class weights to the two types of errors and seeks to ﬁnd a classiﬁer that minimizes the weighted error.
Neyman-Pearson classiﬁcation [Cannon et al., 2002; Scott and Nowak, 2005; Rigollet and Tong, 2011] is another type of method, which aims to minimize one type of error (FNR) subject to a upper bound constraint on the other type of error (FPR).
We call these two types of methods weighted method and constrained method , respectively.
Another popular approach [Fawcett, 2006], which we call cutoff methods , directly constructs a set of classiﬁers based on a discriminant function from some classiﬁer.
Speciﬁcally, the set of classiﬁers is obtained through changing the intercept of the given discriminant function.
Empirically, the cutoff method is computationally efﬁcient and can work well in some situations.
To study the performance of these methods at the population level, i.e., there are inﬁnite number of samples, we ﬁrst introduce a general notion of optimal ROC curve over an arbitrary model space.
This
Systems ( new notion of optimal ROC curve parallels the notion of Bayes error for standard binary classiﬁcation when the model space is the set of all measurable classiﬁers.
Traditionally, ROC curve is typically deﬁned as the set of FPR-TPR pairs produced by the classiﬁers obtained from the cutoff method, where TPR denotes true positive rate and is deﬁned by TPR = 1 − FNR.
An optimal ROC curve [see, e.g., Van Trees, 2004] is deﬁned as the ROC curve generated by η ( x ) =
P ( Y = 1 | X = x ) , where Y denotes the binary response and X denotes the feature vector.
This curve is optimal in the sense that no other ROC curves can lie above it.
Here we generalize the notion of optimal ROC curve given a speciﬁc model space.
This is done by assigning a partial order to all possible FPR-TPR pairs that can be obtained over the given model space, and deﬁning the optimal ROC curve as the pareto frontier of the set of FPR-TPR pairs.
This allows us to deﬁne the optimal ROC curve under an arbitrary model space, and discuss the general case that model is possibly mis-speciﬁed.
We show that the newly deﬁned optimal ROC curve coincides with the existing deﬁnition when the model space includes all possible measurable classiﬁers.
In terms of consistency, we ﬁnd that when the model space contains all measurable classiﬁers, all three methods target the optimal ROC curve under 0 / 1 loss or some surrogate loss under a new notion of correct model speciﬁcation.
Interestingly, in order for a model-loss pair to be correctly speciﬁed, a sufﬁcient condition required for the surrogate loss is related to the notion of classiﬁcation-calibration [Bartlett et al., 2006], which is a sufﬁcient condition required to achieve consistency in binary classiﬁcation.
In this sense, our results generalize that of Bartlett et al. [2006] for binary classiﬁcation to optimal ROC curve estimation.
When the model-loss pair is mis-speciﬁed, however, we show that only the constrained method targets the optimal ROC curve over the chosen model space.
In this sense, the other two methods may miss some optimal FPR-TPR pairs on the optimal ROC curve.
Importantly, this suggests that the constrained method may be preferable to the two other methods in many practical applications where the model space considered does not include the true model.
To the best of our knowledge, our work is the ﬁrst that formally investigate theoretical conditions under which the commonly used methods could recover the optimal ROC curve in a statistical decision theoretical framework.
Moreover, our proposed deﬁnition of optimal ROC curve also generalizes the conventional deﬁnition, relaxing some unnecessary distributional assumptions on the data.
For ROC curve estimation, existing works are either empirical or mostly focus on how to generate a single FPR-TPR pair on the optimal ROC curve using either the weighted method or the constrained method.
For example, Cannon et al.
[2002] investigated the empirical version of the constrained method, and provided a probabilistic guarantee on simultaneous control of both FPR and
FNR.
Extensions to structural risk minimization were later considered in Scott and Nowak [2005].
However, they did not address the issues on how to better estimate the whole ROC curve.
Scott [2007] proposed two families of performance measures to evaluate classiﬁers produced by the constrained method.
Interestingly, they also showed that the ROC curve is concave under some assumptions on the data generating distributions, part of which are special cases of our theory (see Theorem 2).
More recently, Rigollet and Tong [2011] focused on how to obtain a classiﬁer that can control its FPR at the population level.
This is done by using a convex surrogate loss and considering a slightly more stringent upper bound than the desired level for the FPR.
However, their method may not target the best trade-off between FPR and TPR, which is partly attributed to the use of a convex surrogate loss and possible model mis-speciﬁcation.
Lastly, Bach et al.
[2006] proposed to estimate the ROC curve by computing a convex envelope of ROC curves generated by the cutoff method when varying the class weights.
They demonstrate that this will usually outperform the ROC curve produced by the cutoff method alone, which is partly consistent with our theory that the weighted and cutoff method may be sub-optimal when the model is mis-speciﬁed.
Moreover, our deﬁnition of the optimal ROC curve is also new and it extends the old deﬁnition considered by Van Trees [2004] and Scott [2007] in two different ways.
First, the optimal ROC curve in Van Trees [2004] and Scott [2007] requires some continuity assumptions on the data generating distribution.
By contrast, our deﬁnition is quite general and requires a much weaker assumption on the data generating distribution.
In fact, we show that our deﬁnition coincides with the existing deﬁnition if the continuity assumption is satisﬁed.
Second, our deﬁnition allows model mis-speciﬁcation, which reveals some interesting differences among the commonly used methods when model misspeciﬁcation may be present.
Another important contribution is that we formally prove the continuity and concavity of the optimal ROC curve and present some other important properties of the optimal
ROC curve, which extends results informally stated in Van Trees [2004] and Scott [2007].
The rest of the article is structured as follows.
We deﬁne the notion of optimal ROC curve over a general model space, and derive the optimal ROC curve over the entire population analytically in Section 2.
Section 3 analyzes the population analogue of three ROC estimation methods when model is correctly speciﬁed and mis-speciﬁed.
We present some numerical examples to illustrate the impact the mis-speciﬁcation on the performance of three methods in Section 4, and we end with some discussions in Section 5.
All proofs are included in the supplementary material.
Optimal ROC curve over a general model space
In this section, we introduce the notion of optimal ROC curve over a general model space.
The notion of optimal ROC curve is parallel to Bayes risk in standard binary classiﬁcation.
We therefore argue that the optimal ROC curve should be the theoretical target for ROC curve estimation, just like Bayes error in standard binary classiﬁcation.
Before proceeding, we ﬁrst lay out the basic problem setup, some notations, and assumptions to be used throughout the article.
Consider a feature-response pair ( X, Y ) with joint distribution P , where
X ∈ R p is the feature vector and Y ∈ {− 1 , 1 } is the corresponding binary response.
Throughout, we assume that the marginal distribution X , denoted as P X , is continuous, and the marginal distribution of Y satisﬁes p + = P
( Y = 1) ∈ (0 , 1) and p − =
P ( Y = − 1) ∈ (0 , 1) .
We use h : R p
→
{− 1 , 1 } to denote a classiﬁer that maps feature vectors to their responses.
For many methods, a classiﬁer is induced by the sign of a so-called discriminant function .
Throughout, we use H to denote set of classiﬁers, and F to denote set of discriminant functions.
Many different metrics can be used to assess the performance of a classiﬁer, of which misclassiﬁcation error rate P ( Y (cid:54)
=
h ( X )) is perhaps the most commonly used and well-studied in the literature [Zhang et al., 2004; Bartlett et al., 2006].
Classiﬁcation methods are often designed to minimize the misclassiﬁcation error, which is the weighted average of two types of error rate, namely, false positive rate FPR ( h ) =
P ( h ( X ) = 1 | Y = − 1) and false negative rate FNR ( h ) =[-
-]
P ( h ( X ) = − 1 | Y = 1) .
In this article, we focus on the problem of minimizing these two types of error rate simultaneously.
This is essentially equivalent to the estimation of optimal ROC curve, which plots the true positive rate (TPR) against the false positive rate for a set of classiﬁers, where the true positive rate is deﬁned to be TPR ( h ) =
1 − FNR
( h ) .
In standard binary classiﬁcation, the optimal classiﬁer that minimizes the misclassiﬁcation error is often referred to as the Bayes classiﬁer.
For our problem, we have two criterion to optimize.
By borrowing ideas from multi-objective optimization literature [Miettinen, 2012], we deﬁne the set of optimal classiﬁers as those that can not be dominated by any other classiﬁers, and the optimal
ROC curve as the set of FPR-TPR pairs that can not be dominated by any other FPR-TPR pairs.
Here classiﬁer h dominates classiﬁer h (cid:48) , denoted as h (cid:31) h (cid:48) , means that TPR ( h ) ≥ TPR ( h (cid:48) ) ,
FPR ( h ) ≤ FPR ( h (cid:48) ) and ( FPR ( h ) , TPR ( h )) (cid:54) =
( FPR ( h (cid:48) ) , TPR ( h (cid:48) )) .
In this sense, a classiﬁer h is an optimal classiﬁer if it can not be dominated by any other classiﬁers.
Similarly, for any two FPR-TPR pairs ( u, v ) and ( u (cid:48) , v (cid:48) ) , we say that ( u, v ) dominates ( u (cid:48) , v (cid:48) ) , denoted as ( u, v ) (cid:31) ( u (cid:48) , v (cid:48) ) , if u ≤ u (cid:48) , v ≥ v (cid:48) and ( u, v ) (cid:54) =
( u (cid:48) , v (cid:48) ) .
Given this partial ordering, we denote by PF ( A ) :
= { ( u, v ) ∈
A | ( u (cid:48) , v (cid:48) )
(cid:7) ( u, v ) for any ( u (cid:48) , v (cid:48) )
∈ A } the pareto frontier of a compact set A ⊆ R 2 .
Therefore, the set of optimal classiﬁers constitutes the pareto frontier of the given set of classiﬁers, while the optimal ROC curve is the pareto frontier of the given set of FPR-TPR pairs.
Formally, we deﬁne the optimal ROC curve over an arbitrary set of classiﬁers H , which allows possible model mis-speciﬁcation .
Deﬁnition 1 (Optimal ROC curve over model space H )
Given a set of classiﬁers H , denote by
S ( H ) =
{ ( FPR ( h ) , TPR ( h ))
| h ∈ H} the set of all FPR-TPR pairs generated by classiﬁers in H .
We call the pareto frontier of S ( H ) , denoted as γ ( H ) =
PF ( S ( H )) , the optimal receiver operating characteristic (ROC) curve over H .
When H =
H a , where H a is the class of all measurable classiﬁers, we simply call γ ( H a ) the optimal ROC curve.
Throughout this article, we assume that S ( H ) is a closed set so that PF ( S ( H )) is well-deﬁned.
As we can see from Deﬁnition 1, classiﬁers corresponding to any point on the optimal ROC curve cannot be dominated by any other classiﬁer.
Note that this is also similar to the notion of admissibility for general decision rules in statistical decision theory.
The above deﬁnition of optimal ROC curve differs from the existing deﬁnition by Scott [2007] in that it allows for model mis-speciﬁcation.
When the model is correctly speciﬁed, the optimal ROC curve parallels the notion of Bayes error rate for regular binary classiﬁcation, while the corresponding classiﬁers parallels the notion of Bayes classiﬁer.
It is well-known that Bayes error rate is 1 / 2 − E | η ( X ) − 1 / 2 | [Devroye et al., 2013], where η ( x ) =
P ( Y = 1 | X = x ) is the regression function.
In parallel, we derive the optimal ROC curve over the class of all measurable classiﬁers H a .
Compared with some existing deﬁnitions of optimal
ROC curve under correct model speciﬁcation, our deﬁnition is more general, requiring less conditions on the distribution of ( X, Y )
[see, e.g., Scott, 2007].
Speciﬁcally, our deﬁnition of optimal ROC curve only requires that X is a continuous random variable, while Scott [2007] requires that both X and η ( X ) are continuous random variables.
The following theorem derives an analytic form for the optimal ROC curve γ ( H a ) deﬁned in
Deﬁnition 1.
Theorem 1 Let η ( x ) =
P ( Y = 1 | X = x ) , p (cid:63) =
inf { p : P ( η ( X ) ≤ p ) = 1 } , and let I ( · ) denote the indicator function.
We have that the optimal ROC curve can be represented as
[Equation]
where
[Equation]
with c ( α ) =
inf { c ∈ [0 , p (cid:63) ] : E (1 − η ( X ))
( I ( η ( X ) > c ) − α ) ≤ 0 } .
Moreover, s ( α ) is continuous, strictly increasing, and concave over its domain [0 , 1 − P ( η ( X ) = 0)
/p − ] .
Some remarks are in order.
First, we can see that the optimal ROC curve is a continuous, strictly increasing, and concave curve deﬁned by the function s ( α ) over the interval [0 , 1 − P ( η ( X ) = 0) /p − ] .
Second, existing deﬁnition of the optimal ROC curve typically relies on the likelihood ratio test [see, e.g., McIntosh and Pepe, 2002; Scott, 2007].
More speciﬁcally, it assumes that X has classconditional density function f
+ ( x ) and f − ( x ) conditional on Y = 1 and Y = − 1 , and considers a class of classiﬁers indexed by λ ∈ [0 , ∞ ] : { h λ ( x ) =
Sign ( f + ( x ) /f − ( x ) − λ ) , λ ∈ [0 , ∞ ] } .
Then, under the assumption that
[Equation]
for any λ ∈ [0 , ∞ ] , Scott [2007] deﬁned the optimal ROC curve as the set of FPR-TPR pairs generated by the class of classiﬁers { h λ ( x ); λ ∈ [0 , ∞ ] } .
In other words, the existing deﬁnition of the optimal
ROC curve can be expressed as
[Equation]
Our deﬁnition of optimal ROC curve (c.f. Deﬁnition 1), however, does not require the likelihood ratio assumption in (3), and hence in this sense is more general than the existing deﬁnition.
Indeed, assumption(3) may be violated in some common situations, for example, the two class-conditional densities f + ( x ) and f − ( x ) have different supports.
Moreover, under the condition (3), we can show that our proposed deﬁnition coincides with the existing deﬁnition (4).
Furthermore, the optimal ROC curve is the FPR-TPR pairs generated by the set of classiﬁers { Sign( η ( X ) − c ) | c ∈ [0 , 1] } .
We summarize these results in Corollary S1 of the supplementary material.
Lastly, we remark that our proposed notion of the optimal ROC curve (c.f. Deﬁnition 1) also allows model misspeciﬁcation.
This allows us to study the statistical consistency of special classes of classiﬁcation method such as linear classiﬁers.
We will demonstrate in later sections that model speciﬁcation has a meaningful impact on the consistency of various popular classiﬁcation methods for ROC curve estimation.
ROC curve estimation methods
In this section, we review three ROC curve estimation methods that are widely used in practice.
We then investigate, from a statistical decision theoretical point of view, whether these methods are targeting the optimal ROC curve deﬁned in the previous section.
In particular, we shall analyze their population analogue when the model is mis-speciﬁed, that is, the optimal classiﬁer is not included in the set of classiﬁers considered in the model.
3.1 Neyman-Pearson classiﬁcation
Neyman-Pearson classiﬁcation [Scott and Nowak, 2005; Rigollet and Tong, 2011], or the constrained method, minimizes one type of error subject to an upper bound constraint on the other type of error.
The population analogue of Neyman-Pearson classiﬁcation is equivalent to ﬁnding a classiﬁer that maximizes the true positive rate subject to false positive rate constraint, that is, maximize h ∈H
TPR ( h ) subject to FPR ( h ) ≤ α [Equation] (5) where H is a set of classiﬁers.
In this subsection, we will show that under mild conditions the ROC curve generated from the constrained method by varying α ∈ [0 , 1] can recover the optimal ROC curve over general model space H .
Before proceeding, we call the ROC curve generated by the constrained method, deﬁned by
[Equation]
the population ROC curve of the constrained method, where h (cid:63)α, H is a solution to (5).
Note that the population ROC curve is essentially the set of FPR-TPR pairs generated by the solutions to the constrained method (5) after varying α ∈ [0 , 1] .
We ﬁrst consider the case H
= H a , where the model space H a is the set of all measurable classiﬁers.
In this case, we can derive the analytic solution to the constrained method (5), which then allows us to derive the population ROC curve γ C ( H a ) generated by the constrained method.
Theorem 2 Denote η ( x ) =
P ( Y = 1 | X = x ) and p (cid:63) =
inf { p : P ( η ( X ) ≤ p ) = 1 } .
Let H
=
H a be the set of all measurable classiﬁers.
For any solution h (cid:63)α ( x ) of (5) , denote by R (cid:63)α = { x : h (cid:63)α ( x ) = 1 } .
Then, up to a P X -null set,
[Equation]
where c ( α ) = inf { c ∈ [0 , p (cid:63) ] : E (1 − η ( X ))
( I ( η ( X ) > c ) − α ) ≤ 0 } with c ( α ) ∈ [0 , 1) and
N α ⊆ { x : η ( x ) =
c ( α ) } with
[Equation]
Moreover, we have that optimal ROC curve γ ( H a ) satisﬁes
[Equation]
This theorem provides a complete characterization of the population ROC curve of the constrained method.
Combining (8) with Theorem 1, we can see that the constrained method can recover the optimal ROC curve in the sense that it produces a curve that contains the optimal ROC curve.
Using this, we can further show that the pareto frontier of the population ROC curve, deﬁned by γ (cid:63) C ( H a ) =
PF ( cl ( γ C ( H a ))) , is identical to the optimal ROC curve.
Here we use cl ( A ) to denote the closure of set A .
We summarize this result as below.
Corollary 1 We have that
[Equation]
Corollary 1 suggests that the constrained method over H a indeed targets the optimal ROC curve γ ( H a ) .
Essentially, by our proof of Corollary 1, the discrepancy between γ C ( H a ) and the optimal
ROC curve γ ( H a ) occurs only if TPR = 1 .
Moreover, by our proof if P ( η ( X ) = 0)
= {+
 then we+} have that γ C ( H a ) =
γ ( H a ) .
Notably, Scott [2007] also derived the ROC curve generated by the constrained method γ C ( H a ) and proved that it coincides with the optimal ROC curve they deﬁned under likelihood ratio condition (3).
Here, we show that the ROC curve generated by the constrained method coincides with our deﬁned optimal ROC curve (c.f. deﬁnition 1) by only requiring P ( η ( X ) = 0)
= 0 .
Note that for a general model space H , it is no longer possible to derive the population ROC curve analytically.
However, relations like (9) continue to hold.
That is, the pareto frontier of the population
ROC curve for the constrained method, deﬁned by γ (cid:63)C ( H ) =
PF ( cl ( γ C ( H ))) , coincides with the optimal ROC curve over H .
Proposition 1 Suppose that S ( H ) =
{ ( FPR ( h ) , TPR ( h ))
| h ∈ H} is a closed set.
We have that γ (cid:63)C ( H ) =
γ ( H ) ⊆
γ C ( H ) .
Proposition 1 shows that any point on the optimal ROC curve over a general model space H can be recovered by solutions to the constrained method (5) for some α ∈ [0 , 1] .
Later, we shall show that this property is unique to the constrained method for a general model space H .
So far all results consider the 0 / 1 loss.
In practice, a surrogate loss is often used to replace the 0 /loss, and a discriminant function is often used as the decision variable in the optimization problem.
Here we propose to use the so-called psi loss ψ ( x ) = min(1 , max(0 , 1 − x ))
[Shen et al., 2003], which could approximate the 0 / 1 loss up to arbitrary precision.
The rationale here is that a continuous loss that is close to the 0 / 1 loss would enjoy similar nice theoretical properties of the 0 / 1 loss.
As a ﬁnal note, we do not discuss convex surrogate loss for the constrained method, partly because if convex surrogate loss and convex set of discriminant function are used, the optimization problem associated with the constrained method is convex, and thus is equivalent to a weighted problem, which we will study subsequently in the next subsection.
3.2 Weighted method
Another popular method to estimate the ROC curve, which we call the weighted method, considers a set of classiﬁers obtained by assigning different class weights to the two types of errors.
In practice, instead of directly optimizing the “weighted error rate”, some continuous surrogate loss (often convex) is typically used to replace the non-continuous 0 / 1 loss for ease of computation, which could introduce additional “model bias” or model mis-speciﬁcation.
This phenomenon has been extensively studied for standard binary classiﬁcation task [see, e.g., Zhang et al., 2004; Bartlett et al., 2006].
Roughly speaking, the surrogate loss function needs to satisfy certain condition to ensure the consistency of the procedure (in terms of Bayes error).
In Bartlett et al.
[2006], the surrogate loss satisfying this condition is called to be classiﬁcation-calibrated [Bartlett et al., 2006], and they further showed that procedures that are consistent using the 0 / 1 loss, are also consistent using any classiﬁcation-calibrated losses.
Here we shall present some parallel results for ROC curve estimation.
More speciﬁcally, we consider the population analogue of the weighted method under a surrogate loss V ( · ) over a set of discriminant functions F :
[Equation]
We denote by f (cid:63)w,V ( · ) a solution to (10), which may not necessarily be unique.
Given the optimal discriminant function f (cid:63)w,V ( · ) , we deﬁne the associated optimal classiﬁer by h (cid:63)w,V ( · ) =
Sign( f (cid:63)w,V ( · ) ) , where Sign ( z ) =
I ( z ≤ 0) − 1 denotes the sign function.
We aim to ﬁnd conditions for the surrogate loss and the class of discriminant function, under which the population ROC curve of the weighted method using a surrogate loss V ( · ) over a set of discriminant functions F , deﬁned as
[Equation]
can recover the optimal ROC curve over H =
{ Sign( f ) | f ∈ F} .
We ﬁrst present a general condition on the class of discriminant functions F and the surrogate loss function V ( · ) to ensure consistent estimation of the optimal ROC curve.
Deﬁnition 2 (Correct model speciﬁcation for weighted method)
We say that a set of discriminant function F is correctly speciﬁed under a surrogate loss V ( · ) for the weighted method, if for any 0 ≤ w ≤ 1 with P ( η ( X ) = w ) = 0 , any solution f (cid:63)w,V ( · ) to (10) satisﬁes { x : f (cid:63)w,V ( · ) ( x ) > 0 } = { x : η ( x ) > w } up to a P X -null set.
Under this notion of correct model speciﬁcation, we can indeed show that the weighted method using the surrogate loss V ( · ) recovers the optimal ROC curve if the model space F is correctly speciﬁed under V ( · ) .
Theorem 3 If F is correctly speciﬁed under a surrogate loss V ( · ) for the weighted method, then we have that
[Equation]
is non-differentiable at α, α ∈
[Equation]
[Equation]
1 − P
[Equation]
η
[Equation]
X
[Equation]
= [Equation] p − )
[Equation]
∪
[Equation]
where S (cid:48) 0 and S (cid:48) 1 satisfy
{+S (cid:48) 
 =+}
[Equation] [Equation]
0 , s
[Equation]
[Equation]
} , if c (0) < 1 or P ( η ( X ) = 1) = 0 , ∅ , otherwise , (13) and
S
[Equation]
, if
[Equation]
otherwise .
Theorem 3 connects the population ROC curve of the weighted method to the optimal ROC curve when H =
H a .
In Proposition S1 of the supplementary material, we establish that the set of ( α, s ( α )) with P ( η ( X ) =
c ( α )) > 0 and α > 0 corresponds to linear pieces on γ ( H a ) .
Combining this with (12) in Theorem 3, it follows that except for some interior points of these linear pieces the other parts over optimal ROC curve can be recovered by weighted method.
Therefore, roughly speaking, the set of points on the optimal ROC curve that can be recovered by the weighted method is the union of non-differentiable part and nonlinear part (up to some isolated points).
Although the above result seems to suggest that the weighted method may not recover all these linear pieces on the optimal ROC curve, it can be trivially modiﬁed to recover the entire optimal ROC curve by considering the pareto frontier of the convex hull of the ROC curve generated by weighted method.
This is established in Corollary S2 of the supplementary material.
Next, we establish sufﬁcient conditions on V ( · ) under which F a is correctly speciﬁed under V ( · ) , where F a consists of all measurable functions.
Interestingly, the requirement on V ( · ) coincides with the notion of classiﬁcation-calibrated [Bartlett et al., 2006].
Deﬁnition 3 (Classiﬁcation-calibrated loss [Bartlett et al., 2006]) Let 0 ≤ η ≤ 1 and deﬁne
[Equation] [Equation]
(16) where V ( · ) is a surrogate loss function.
V ( · ) is said to be classiﬁcation-calibrated if H V ( η ) <
H − V ( η ) for any η (cid:54) = 1 / 2 .
In Bartlett et al.
[2006], it is shown that using a classiﬁcation-calibrated loss V ( · ) ensures consistent estimation of the optimal misclassiﬁcation error, when F includes all measurable function (i.e.,
F = F a ).
Now we establish that the class of all measurable discriminant functions F a is correctly speciﬁed under any classiﬁcation-calibrated surrogate loss.
Proposition 2 The class of all measurable functions F a is correctly speciﬁed for the weighted method (i.e., in the sense of Deﬁnition 2) under the 0 / 1 loss, and any classiﬁcation-calibrated surrogate loss
V ( · ) .
Combining this with Theorem 3, it follows that if V ( · ) is classiﬁcation-calibrated and F = F a , then the weighted method using the surrogate loss can recover the optimal ROC curve.
In other words, for weighted method using classiﬁcation-calibrated loss over F = F a generates consistent estimation of the optimal ROC curve.
Finally, we give an example in Theorem S1 of the supplementary material to show that in contrast to the constrained method (c.f. Proposition 1), the population ROC curve of the weighted method may miss some components of optimal ROC curve when the model space F is mis-speciﬁed under a surrogate loss V ( · ) .
Importantly, these components cannot be recovered even after taking the convex hull transformation.
3.3 Cutoff method
Cutoff method is another type of methods commonly used in practice.
It ﬁrst produces a discriminant function using some classiﬁcation method, and then generates a set of classiﬁers by thresholding a sequence of cutoffs to the discriminant function.
In order to study the population analogue of the cutoff method, we assume that the discriminant function is generated from a given classiﬁcation method with class weight w ∈ [0 , 1] for negative instances:
[Equation]
where F is the set of discriminant functions.
We deﬁne the population ROC curve of the cutoff method over F to be
[Equation]
Next, we establish sufﬁcient conditions on F and V ( · ) to ensure that the cutoff method can recover the optimal ROC curve.
Similar to the weighted method, we deﬁne a notion of correct model speciﬁcation for the cutoff method.
Deﬁnition 4 (Correct model speciﬁcation for cutoff method)
We say that a set of discriminant function F is correctly speciﬁed under a surrogate loss function V ( · ) for the cutoff method for some w ∈ (0 , 1) , if any solution to (17) at w is a strictly increasing transformation to η ( x ) almost surely, that is,
[Equation]
surely , where M ( · ) is a strictly increasing function.
Under this notion of correct model speciﬁcation, we show that the cutoff method targets the optimal
ROC curve when η ( X ) is a continuous random variable.
This is established in the following theorem.
Theorem 4 Assume η ( X ) is a continuous random variable.
If F is correctly speciﬁed under a surrogate loss V ( · ) for the cutoff method for some w ∈ [0 , 1] , then we have that
[Equation]
For the cutoff method, we can show that the set of all measurable discriminant functions is correctly speciﬁed under any surrogate loss V ( · ) that is differentiable, strictly decreasing, and strictly convex.
Theorem 5 Assume η ( X ) is a continuous random variable.
Suppose that V ( · ) is a differentiable, strictly decreasing, proper and strictly convex loss.
Then the class of all measurable functions F a is correctly speciﬁed under V ( · ) for the cutoff method at any w ∈ (0 , 1) .
An important implication of the above two theorems is that, the cutoff method deﬁned in (18) can recover the optimal ROC curve if F = F a and the surrogate loss V ( · ) is differentiable, strictly decreasing and strictly convex.
Note that the requirement on the surrogate loss for the model to be correctly speciﬁed for the cutoff method is more stringent than the classiﬁcation-calibrated requirement for the weighted method (c.f., Lemma S7 in the supplementary material).
For a general model space F , the requirement of correct model speciﬁcation for the cutoff method is also generally stronger than those needed for the weighted method.
This is because in general we would not know whether f (cid:63)w,V ( · ) ( x ) is an strictly increasing function of η ( x ) ,
even if
Sign( f (cid:63)w,V ( · ) ( x ))
=
Sign( η ( x ) − w ) and η ( X ) is a continuous random variable.
In Theorem S2 of the supplementary material, we give an example to show that in contrast to the constrained method (c.f. Proposition 1), the population ROC curve generated by the cutoff method over F may be dominated by the optimal ROC curve over H =
{ Sign ( f − δ ) | f ∈ F , δ ∈ R } , suggesting that the cutoff method may not recover the optimal ROC curve over H in general.
Simulations and real data analysis
In this section, we examine the operating characteristics of the three methods analyzed in this article with both simulation studies and a real data example.
In our simulation, we investigate the effect of model mis-speciﬁcation by considering linear classiﬁers over two simulated data sets.
One dataset is generated under a linear discriminant analysis (LDA) setting, which imitates the scenario where model is correctly speciﬁed.
The other is based on a quadratic discriminant analysis (QDA) setting, which is used to show the differences across three methods under model mis-speciﬁcation.
We also compare the three methods by using a bank marketing data set [Moro et al., 2014], which allows us to investigate the effect model mis-speciﬁcation on the performance of ROC curve estimation methods in real problem setting.
Throughout all the experiments, we consider linear classiﬁcation methods including constrained ψ -learning, weighted SVM and its cutoff version, and also typical kernel methods that include kernel weighted SVM and its cutoff version
Example 1 (LDA setting).
This example considers a linear discriminant analysis model, where P ( Y = 1) =
P ( Y = − 1) = 1 / 2 , and conditioned on Y , X | Y ∼ N ( Y µ , 4 I ) , where µ = (1 , 1) (cid:62) ∈ R 2 .
The sample size is n = 2000 .
Example 2 (QDA setting).
This example considers a quadratic discriminant analysis model, where P ( Y = 1) =
P ( Y = − 1) = 1 / 2 , and conditioned on Y , X | Y ∼ N ( Y µ , Σ Y ) , where µ = (1 , 1) (cid:62) ∈ R 2 , and Σ + = (cid:18) 85(cid:19) , Σ − = (cid:18)−− (cid:19) .
The sample size is n = 2000 .
Example 3 (Real data example).
This example considers a bank marketing dataset [Moro et al., 2014], which records the direct marketing campaigns of a Portuguese banking institution and are available at https://archive.ics.uci.edu/ml/datasets/Bank+Marketing .
This real dataset consists of 4521 examples with 17 features and 1 binary response variable.
We visualize the scatter plots of two simulation examples in Figure 1.
For two simulation examples, we compare the average estimated ROC curve for the ﬁve aforementioned methods over 100 replicates; while for the real data example, we present the estimated ROC curves directly.
For all three examples, to generate the estimated ROC curve we vary weight w and the constraint upper bound α from { i/ 500 | i = 0 , 1 , . . . , 500 } for the weighted method and the constrained method, respectively.
For the two cutoff methods, we choose w = 1 / 2 .
The results are summarized in Figure 2 and Figure S of the supplementary material.
The left panel of Figure 2 is for the simulations under LDA setting, where the model is correctly speciﬁed for all methods.
In this case, all methods perform almost the same.
The right panel of Figure 2 is the for the simulations under QDA setting.
In this case, the model is mis-speciﬁed for linear methods.
As expected, the two kernel methods perform the best.
Moreover, among linear methods, the constrained method performs the best, which corroborates with our theoretical ﬁndings that constrained method performs best under possible model mis-speciﬁcation.
For our real data example, we can see in Figure S1 that among all these linear method the constrained method performs best, but is no better than the weighted kernel SVM.
These results again corroborate with our theoretical analysis on the goodness of using constrained method when model is mis-speciﬁed.
Discussion
In this article, we deﬁne a general notion of optimal ROC curve, which we argue should be the theoretical target of any ROC curve estimation methods.
We analyze three popular ROC estimation methods, and show that when the model space contains all measurable discriminant functions all three methods target the optimal ROC curve, although the cutoff method requires a stronger set of conditions on the surrogate loss for it to be consistent.
When model mis-speciﬁcation is present, we show that only the constrained method under the 0 / 1 loss targets the optimal ROC curve over the used model space.
This together with our simulations and real data studies suggests that the constrained method with a non-convex surrogate loss (e.g. ψ loss) would be the preferred method when the model is possibly mis-speciﬁed.
Although we have observed good empirical performances using the constrained method with ψ loss, we have not investigated its theoretical guarantees in this article.
Further theoretical research on non-convex surrogate loss to achieve guaranteed performance may be necessary.
[Figure]
Acknowledgments and Disclosure of Funding
The authors are supported in part by the US National Science Foundation (NSF) under grant DMS20-15490.
The authors thank the reviewers for valuable comments and suggestions that improved the article.