On the consistent estimation of optimal Receiver Operating Characteristic (ROC) curve
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Under a standard binary classiﬁcation setting with possible model misspeciﬁcation,we study the problem of estimating general Receiver Operating Characteristic(ROC) curve, which is an arbitrary set of false positive rate (FPR) and true positiverate (TPR) pairs.
We formally introduce the notion of optimal ROC curve overa general model space.
It is argued that any ROC curve estimation methodsimplemented over the given model space should target the optimal ROC curve overthat space.
Three popular ROC curve estimation methods are then analyzed atthe population level under both correct and incorrect model speciﬁcation.
Basedon our analysis, they are all consistent when the surrogate loss function satisﬁescertain conditions and the given model space includes all measurable classiﬁers.
Interestingly, some of these conditions are similar to those that are required to ensure classiﬁcation consistency.
When the model space is incorrectly speciﬁed,however, we show that only one method leads to consistent estimation of the ROC curve over the chosen model space.
We present some numerical results todemonstrate the effects of model misspeciﬁcation on the performance of variousmethods in terms of their ROC curve estimates.
Introduction
One common goal in standard binary classiﬁcation task is to ﬁnd a classiﬁer with minimum misclassiﬁcation error, which is the average of the false negative rate (FNR) and false positive rate(FPR).
This implicitly assumes that the two types of errors are equally important.
In many realapplications, however, these two types of errors may need to be treated unequally by assigningasymmetric losses to them.
For example, in spam email detection, classifying non-spam as spam mayincur a higher cost than classifying spam into non-spam.
Existing methods to handle this problemcan be roughly categorized into three groups.
Cost-sensitive learning [Elkan, 2001] assigns unequalclass weights to the two types of errors and seeks to ﬁnd a classiﬁer that minimizes the weightederror.
Neyman-Pearson classiﬁcation [Cannon et al., 2002; Scott and Nowak, 2005; Rigollet and Tong, 2011] is another type of methods, which aims to minimize one type of error (FNR) subject to aupper bound constraint on the other type of error (FPR).
We call these two types of methods weightedmethod and constrained method , respectively.
Another popular approach [Fawcett, 2006], which wecall cutoff methods , directly constructs a set of classiﬁers based on a discriminant function from someclassiﬁer.
Speciﬁcally, the set of classiﬁer is obtained through changing the intercept of the givendiscriminant function.
Empirically, the cutoff method is computationally efﬁcient and can work wellin some situations.
To study the performance of these methods at the population level, we ﬁrst introduce a general notionof optimal ROC curve over an arbitrary model space.
This new notion of optimal ROC curve parallelsthe notion of Bayes error for standard binary classiﬁcation when the model space is the set of all measurable classiﬁers.
Traditionally, ROC curve is typically deﬁned as the set of FPR-TPR pairsproduced by the classiﬁers obtained from the cutoff method, where TPR denotes true positive rateand is deﬁned by TPR = 1 − FNR.
An optimal ROC curve [see, e.g., Van Trees, 2004] is deﬁned asthe ROC curve generated by η ( x ) =
P ( Y = 1 | X = x ) , where Y denotes the binary response and X denotes the feature vector.
This curve is optimal in the sense that no other ROC curves can lieabove it.
Here we generalize the notion of optimal ROC curve given a speciﬁc model space.
Thisis done by assigning a partial order to all possible FPR-TPR pairs that can be obtained over thegiven model space, and deﬁning the optimal ROC curve as the pareto frontier of the set of FPR-TPRpairs.
This allows us to deﬁne the optimal ROC curve under an arbitrary model space.
We show thatthe newly deﬁned optimal ROC curve coincides with the existing deﬁnition when the model spaceincludes all possible measurable classiﬁers.
In terms of consistency, we ﬁnd that when the model space contains all measurable classiﬁers, allthree methods targets the optimal ROC curve under 0 / 1 loss or some surrogate loss under a newnotion of correct model speciﬁcation.
Interestingly, in order for a model-loss pair to be correctlyspeciﬁed, a sufﬁcient condition required for the surrogate loss is related to the notion of classiﬁcationcalibration [Bartlett et al., 2006], which is a sufﬁcient condition required to achieve consistency inbinary classiﬁcation.
In this sense, our results generalize that of [Bartlett et al., 2006] for binaryclassiﬁcation to optimal ROC curve estimation.
When the model-loss pair is mis-speciﬁed, however,we show that only the constrained method targets the optimal ROC curve over the chosen modelspace.
In this sense, the other two methods may miss some optimal FPR-TPR pairs on the optimal
ROC curve.
Importantly, this suggests that the constrained method may be preferable to the two othermethods when simple classiﬁers (say linear classiﬁers) are considered.
To the best of our knowledge, our work is the ﬁrst that formally investigate theoretical conditionsunder which the commonly used methods could recover the optimal ROC curve in a statisticaldecision theoretical framework.
Moreover, our proposed deﬁnition of optimal ROC curve alsogeneralizes the conventional deﬁnition, relaxing some unnecessary distributional assumptions onthe data.
For ROC curve estimation, existing works are either empirical or mostly focuses on howto generate a single FPR-TPR pair on the optimal ROC curve using either the weighted method orthe constrained method.
For example, Cannon et al.
[2002] investigated the empirical version of theconstrained method, and provided a probabilistic guarantee on simultaneous control of both FPR and
FNR.
Extensions to structural risk minimization were later considered in Scott and Nowak [2005].
However, they did not address the issues on how to better estimate the whole ROC curve.
Scott [2007]proposed two families of performance measures to evaluate classiﬁers produced by the constrainedmethod.
Interestingly, they also showed that the ROC curve is concave under some assumptions onthe data generating distributions, part of which are special cases of our theory (see Theorem 2).
Morerecently, Rigollet and Tong [2011] focuses on how to obtain a classiﬁer that can control its FPR atthe population level.
This is done by using a convex surrogate loss and considering a slightly morestringent upper bound than the desired level for the FPR.
However, their method may not target thebest trade-off between FPR and TPR, which is partly attributed to the use of a convex surrogate lossand possible model mis-speciﬁcation.
Lastly, Bach et al.
[2006] proposed to estimate the ROC curveby computing a convex envelope of ROC curves generated by the cutoff method when varying theclass weights.
They demonstrate that this will usually outperform the ROC curve produced by thecutoff method alone, which is partly consistent with our theory that the weighted and cutoff methodmay be sub-optimal when the model is mis-speciﬁed.
Moreover, our deﬁnition of the optimal ROC curve is also new and it extends the old deﬁnitionconsidered by Van Trees [2004]; Scott [2007] in two different ways.
First, the optimal ROC curvein [Van Trees, 2004; Scott, 2007] requires some continuity assumptions on the data generatingdistribution.
However, our deﬁnition is quite general and does not make any assumptions on the datagenerating distribution.
In fact, we show that our deﬁnition coincides with the existing deﬁnitionif the continuity assumption is satisﬁed.
Second, our deﬁnition allows model mis-speciﬁcation,which reveals some interesting differences among the commonly used methods when model misspeciﬁcation may be present.
Another important contribution is that we formally prove the continuityand concavity of the optimal ROC curve and present some other important properties of the optimal
ROC curve, which extends results informally stated in [Van Trees, 2004; Scott, 2007].
The rest of the article is structured as follows.
We deﬁne the notion of optimal ROC curve overa general model space, and derive the optimal ROC curve over the entire population analyticallyin Section 2.
Section 3 analyzes the population analogue of three ROC estimation methods when model is correctly speciﬁed and mis-speciﬁed.
We present some numerical examples to illustratethe impact the mis-speciﬁcation on the performance of three methods in Section 4, and we end withsome discussions in Section 5.
All proofs are included in appendix.
Optimal ROC curve over a general model space
In this section, we introduce the notion of optimal ROC curve over a general model space.
The notionof optimal ROC curve is parallel to Bayes risk in standard binary classiﬁcation.
We therefore arguethat the optimal ROC curve should be the theoretical target for ROC curve estimation, just like Bayeserror in standard binary classiﬁcation.
Before proceeding, we ﬁrst lay out the basic problem setup, some notations, and assumptions to beused throughout the article.
Consider a feature-response pair ( X, Y ) with joint distribution P , where
X ∈ R p is the feature vector and Y ∈ {− 1 , 1 } is the corresponding binary response.
Throughout,we make the assumption that the marginal distribution X , denoted as P X , is continuous, and themarginal distribution of Y satisﬁes p + = P
( Y = 1) ∈ (0 , 1) and p − =
P ( Y = − 1) ∈ (0 , 1) .
Weuse h :
R p
→
{− 1 , 1 } to denote a classiﬁer that maps feature vectors to their responses.
For manymethods, a classiﬁer is induced by the sign of a so-called discriminant function .
Throughout, we use
H to denote set of classiﬁers, and F to denote set of discriminant functions.
Many different metrics can be used to assess the performance of a classiﬁer, of which misclassiﬁcationerror rate P ( Y (cid:54)
= h ( X )) is perhaps the most commonly used and well-studied in the literature[Zhang et al., 2004; Bartlett et al., 2006].
Classiﬁcation methods are often designed to minimizethe misclassiﬁcation error, which is the weighted average of two types of error rate, namely, falsepositive rate (FPR): FPR ( h ) =
P ( h ( X ) = 1 | Y = − 1) and false negative rate (FNR): FNR ( h ) =[-
-]
P h ( X )
= − 1 | Y = 1) .
In this article, we focus on the problem of minimizing these two typesof error rate simultaneously.
This is essentially equivalent to the optimal estimation of ROC curve,which plots the true positive rate (TPR) against the false positive rate for a set of classiﬁers, wherethe true positive rate is deﬁned to be TPR ( h )
= 1 − FNR
( h ) .
In standard binary classiﬁcation, the optimal classiﬁer that minimizes the misclassiﬁcation error isoften referred to as the Bayes classiﬁer.
For our problem, we have two criterion to optimize.
Byborrowing ideas from multi-objective optimization literature [Miettinen, 2012], we deﬁne the setof optimal classiﬁers as those that can not be dominated by any other classiﬁers, and the optimal
ROC curve as the set of FPR-TPR pairs that can not be dominated by any other FPR-TPR pairs.
Here classiﬁer h dominates classiﬁer h (cid:48) , denoted as h (cid:31) h (cid:48) , means that TPR ( h ) ≥ TPR ( h (cid:48) ) ,
FPR ( h ) ≤ FPR ( h (cid:48) ) and ( FPR ( h ) , TPR ( h )) (cid:54) =
( FPR ( h (cid:48) ) , TPR ( h (cid:48) )) .
In this sense, a classiﬁer h is anoptimal classiﬁer if it can not be dominated by any other classiﬁers.
Similarly, for any two FPR-TPRpairs ( u, v ) and ( u (cid:48) , v (cid:48) ) , we say that ( u, v ) dominates ( u (cid:48) , v (cid:48) ) , denoted as ( u, v ) (cid:31) ( u (cid:48) , v (cid:48) ) , if u ≤ u (cid:48) ,v ≥ v (cid:48) and ( u, v ) (cid:54) =
( u (cid:48) , v (cid:48) ) .
Given this partial ordering, we denote by PF ( A ) :
= { ( u, v ) ∈
A |( u (cid:48) , v (cid:48) ) (cid:7) ( u, v ) for any ( u (cid:48) , v (cid:48) )
∈ A } the pareto frontier of a compact set A ⊆ R 2 .
Therefore, theset of optimal classiﬁers constitutes the pareto frontier of the set of all measurable classiﬁers, whilethe optimal ROC curve is the pareto frontier of the set of all possible FPR-TPR pairs.
Formally, we deﬁne the optimal ROC curve over an arbitrary set of classiﬁers H , which allowspossible model mis-speciﬁcation .
Deﬁnition 1 (Optimal ROC curve over model space H )
Given a set of classiﬁer H , denote by
S ( H ) =
{ ( FPR ( h ) , TPR ( h ))
| h ∈ H} the set of all FPR-TPR pairs generated by classiﬁers in H .
We call the pareto frontier of S ( H ) , denoted as γ ( H ) =
PF ( S ( H )) , the optimal receiver operatingcharacteristic (ROC) curve over H .
When H =
H a , where H a is the class of all measurableclassiﬁers, we simply call γ ( H a ) the optimal ROC curve.
Throughout this article, we assume that S ( H ) is a closed set so that PF ( S ( H )) is well-deﬁned.
Aswe can see from Deﬁnition 1, classiﬁers corresponding to any point on the optimal ROC curve cannotbe dominated by any other classiﬁer.
Note that this is also similar to the notion of admissibilityfor general decision rules in statistical decision theory.
The above deﬁnition of optimal ROC curvediffers from the existing deﬁnition by Scott [2007] in that it allows for model mis-speciﬁcation.
When the model is correctly speciﬁed, the optimal ROC curve parallels the notion of Bayes errorrate for regular binary classiﬁcation, while the corresponding classiﬁers parallels the notion of Bayesclassiﬁer.
It is well-known that Bayes error rate is 1 / 2 − E | η ( X ) − 1 / 2 | [Devroye et al., 2013], whereη ( x ) =
P ( Y = 1 | X = x ) is the regression function.
In parallel, we derive the optimal ROC curve over the class of all measurable classiﬁers H a .
Compared with some existing deﬁnitions of optimal
ROC curve under correct model speciﬁcation, our deﬁnition is more general, requiring less conditionson the distribution of ( X, Y )
[see, e.g., Scott, 2007].
Speciﬁcally, our deﬁnition of optimal ROCcurve only requires that X is a continuous random variable, while Scott [2007] requires that both Xand η ( X ) are continuous random variables.
The following theorem derives an analytic form for the optimal ROC curve γ ( H a ) deﬁned in
Deﬁnition 1.
Theorem 1 Let η ( x ) =
P ( Y = 1 | X = x ) and p (cid:63) = inf { p : P ( η ( X ) ≤ p ) = 1 } .
We have that theoptimal ROC curve can be represented as
[Equation]
wheres ( α ) =
(cid:40) P ( η ( X ) = 1)
/p + if α = 0 ,
[Equation]
with c ( α ) =
inf { c ∈ [0 , p (cid:63) ] : E (1 − η ( X ))
( I ( η ( X ) > c ) − α ) ≤ 0 } .
Moreover, s ( α ) is continuous,strictly increasing, and concave over its domain [0 , 1 − P ( η ( X ) = 0)
/p − ] .
Some remarks are in order.
First, we can see that the optimal ROC curve is a continuous, strictly increasing, and concave curve deﬁned by the function s ( α ) over the interval [0 , 1 − P ( η ( X ) = 0) /p − ] .
Second, existing deﬁnition of the optimal ROC curve typically relies on the likelihood ratio test[see, e.g., McIntosh and Pepe, 2002; Scott, 2007].
More speciﬁcally, it assumes that X has classconditional density function f
+ ( x ) and f − ( x ) conditional on Y = 1 and Y = − 1 , and considers aclass of classiﬁers indexed by λ ∈ [0 , ∞ ] : { h λ ( x ) =
Sign ( f + ( x ) /f − ( x ) − λ ) , λ ∈ [0 , ∞ ] } .
Then,under the assumption that
[Equation]
for any λ ∈ [0 , ∞ ] , Scott [2007] deﬁned the optimal ROC curve as the set of FPR-TPR pairs generatedby the class of classiﬁers { h λ ( x ); λ ∈ [0 , ∞ ] } .
In other words, the existing deﬁnition of the optimal
ROC curve can be expressed as
[Equation]
Our deﬁnition of optimal ROC curve (c.f. Deﬁnition 1), however, does not require the likelihoodratio assumption in (3), and hence in this sense is more general than the existing deﬁnition.
Indeed,assumption(3) may be violated in some common situations, for example, the two class-conditionaldensities f + ( x ) and f − ( x ) have different supports.
Moreover, under the condition (3), we can show that our proposed deﬁnition coincides with theexisting deﬁnition (4).
Furthermore, the optimal ROC curve is the FPR-TPR pairs generated by theset of classiﬁers { Sign( η ( X ) − c ) | c ∈ [0 , 1] } .
These are summarized in the following corollary. Corollary 1 Assume that (3) holds. Then [Equation]
This result has also been established in McIntosh and Pepe [2002] by implicitly assuming condition(3). Moreover, Scott [2007] also established the concavity of the optimal ROC curve under theaforementioned assumptions (3). Here, we establish in Theorem 1 that the optimal ROC curve isalways continuous, strictly increasing, and concave, without assuming (3). We also include someadditional properties regarding the linearity and differentiability of the optimal ROC curve in Section
C of supplementary material.
Lastly, we remark that our proposed notion of the optimal ROC curve (c.f. Deﬁnition 1) alsoallows model misspeciﬁcation.
This allows us to study the statistical consistency of special classof classiﬁcation method such as linear classiﬁers.
We will demonstrate in later sections that modelspeciﬁcation has a meaningful impact on the consistency of various popular classiﬁcation methodsfor ROC curve estimation.
ROC curve estimation methods
In this section, we review three ROC curve estimation methods that are widely used in practice.
We then investigate, from a statistical decision theoretical point of view, whether these methods aretargeting the optimal ROC curve deﬁned in the previous section.
In particular, we shall analyze theirpopulation analogue when the model is possibly mis-speciﬁed.
3.1 Neyman-Pearson classiﬁcation
Neyman-Pearson classiﬁcation [Scott and Nowak, 2005; Rigollet and Tong, 2011], or the constrainedmethod, minimizes one type of error subject to an upper bound constraint on the other type of error.
The population analogue of Neyman-Pearson classiﬁcation is equivalent to ﬁnding a classiﬁer thatmaximizes the true positive rate subject to false positive rate constraint, that is,
[Equation] where H is a set of classiﬁers.
In this subsection, we will show that under mild conditions the ROCcurve generated from the constrained method by varying α ∈ [0 , 1] can recover the optimal ROCcurve over general model space H .
Before proceeding, we call the ROC curve generated by the constrained method, deﬁned by
[Equation]
the population ROC curve of the constrained method, where h (cid:63)α, H is a solution to (6).
Note thatthe population ROC curve is essentially the set FPR-TPR pairs generated by the solutions to theconstrained method (6) after varying α ∈ [0 , 1] .
We ﬁrst consider the case H
= H a , where the model space H a is the set of all measurable classiﬁers.
In this case, we can derive the analytic solution to the constrained method (6), which then allows usto derive the population ROC curve γ C ( H a ) generated by the constrained method.
Theorem 2 Denote η ( x ) =
P ( Y = 1 | X = x ) and p (cid:63) = inf { p : P ( η ( X ) ≤ p ) = 1 } .
Let H
=
H abe the set of all measurable classiﬁers.
For any solution h (cid:63)α ( x ) of (6) , denote by R (cid:63)α = { x : h (cid:63)α ( x ) =1 } .
Then, up to a P X -null set,
[Equation]
where c ( α ) = inf { c ∈ [0 , p (cid:63) ] : E (1 − η ( X ))
( I ( η ( X ) > c ) − α ) ≤ 0 } with c ( α ) ∈ [0 , 1) and
N α ⊆ { x : η ( x ) =
c ( α ) } with
[Equation]
Moreover, we have that optimal ROC curve γ ( H a ) satisﬁes
[Equation]
This theorem provides a complete characterization of the population ROC curve of the constrainedmethod.
Combining (9) with Theorem 1, we can see that the constrained method can recover theoptimal ROC curve in the sense that it produces a curve that contains the optimal ROC curve.
Using this, we can further show that the pareto frontier of the population ROC curve, deﬁned byγ (cid:63)C ( H a ) =
PF ( cl ( γ C ( H a ))) , is identical to the optimal ROC curve, which is summarized below.
Corollary {+
+} We have that
[Equation]
Corollary 2 suggests that the constrained method over H a indeed targets the optimal ROC curveγ ( H a ) .
Essentially, by our proof of Corollary 2, the discrepancy between γ C ( H a ) and the optimal
ROC curve γ ( H a ) occurs only if TPR = 1 .
Moreover, by our proof if P ( η ( X ) = 0)
= 0 thenwe have that γ C ( H a ) =
γ ( H a ) .
Notably, Scott [2007] also derived the ROC curve generated by the constrained method γ C ( H a ) and proved that it coincides with the optimal ROC curve theydeﬁned under likelihood ratio condition (3).
Here, we show that the ROC curve generated by theconstrained method coincides with our deﬁned optimal ROC curve (c.f. deﬁnition 1) by only requiring 227 P ( η ( X ) = 0)
= 0 .
Note that for a general model space H , it is no longer possible to derive the population ROCcurve analytically.
However, relations like (10) continue to hold.
That is, the pareto frontier of thepopulation ROC curve for the constrained method, deﬁned by γ (cid:63)C ( H ) =
PF ( cl ( γ C ( H ))) , coincideswith the optimal ROC curve over H .
Proposition 1 Suppose that S ( H ) =
{ ( FPR ( h ) , TPR ( h ))
| h ∈ H} is a closed set.
We have thatγ (cid:63)C ( H ) =
γ ( H ) ⊆
γ C ( H ) .
Proposition 1 shows that any point on the optimal ROC curve over a general model space H can berecovered by solutions to the constrained method (6) for some α ∈ [0 , 1] .
Later, we shall show thatthis property is unique to the constrained method for a general model space H .
So far all results consider the 0 / 1 loss.
In practice, a surrogate loss is often used to replace the 0 / 1loss, and a discriminant function is often used as the decision variable in the optimization problem.
Here we propose to use the so-called psi loss ψ ( x ) = min(1 , max(0 , 1 − x ))
[Shen et al.
, 2003],which could approximate the 0 / 1 loss up to arbitrary precision.
The rationale here is that a continuousloss that is close to the 0 / 1 loss would enjoy similar nice theoretical properties of the 0 / 1 loss.
Asa ﬁnal note, we do not discuss convex surrogate loss for the constrained method, partly because ifconvex surrogate loss and convex set of discriminant function are used, the optimization problemassociated with the constrained method is convex, and thus is equivalent to a weighted problem,which we will study subsequently in the next subsection.
3.2 Weighted method
Another popular method to estimate the ROC curve, which we call the weighted method, considers aset of classiﬁers obtained by assigning different class weights to the two types of errors.
In practice,instead of directly optimizing the “weighted error rate”, some continuous surrogate loss (oftenconvex) is typically used to replace the non-continuous 0 / 1 loss for ease of computation, whichcould introduce additional “model bias” or model mis-speciﬁcation.
This phenomenon has beenextensively studied for standard binary classiﬁcation task [see, e.g., Zhang et al., 2004; Bartlett et al.
Roughly speaking, the surrogate loss function needs to satisfy certain condition to ensurethe consistency of the procedure (in terms of Bayes error).
In Bartlett et al.
[2006], the surrogateloss satisfying this condition is called to be classiﬁcation-calibrated [Bartlett et al., 2006], and theyfurther showed that procedures that are consistent using the 0 / 1 loss, are also consistent using anyclassiﬁcation-calibrated losses.
Here we shall present some parallel results for ROC curve estimation.
More speciﬁcally, we considerthe population analogue of the weighted method under a surrogate loss V ( · ) over a set of discriminantfunctions F :
[Equation]
We denote by f (cid:63)w,V ( · ) a solution to (11), which may not necessarily be unique.
Given the optimaldiscriminant function f (cid:63)w,V ( · ) , we deﬁne the associated optimal classiﬁer by h (cid:63)w,V ( · ) =
Sign( f (cid:63)w,V ( · ) ) ,where Sign ( z ) =
I ( z ≤ 0) − 1 denotes the sign function.
We aim to ﬁnd conditions for the surrogateloss and the class of discriminant function, under which the population ROC curve of the weightedmethod using a surrogate loss V ( · ) over a set of discriminant functions F , deﬁned as
[Equation]
can recover the optimal ROC curve over H =
{ Sign( f ) | f ∈ F} .
We ﬁrst present a generalcondition on the class of discriminant functions F and the surrogate loss function V ( · ) to ensureconsistent estimation of the optimal ROC curve.
Deﬁnition 2 (Correct model speciﬁcation for weighted method)
We say that a set of discriminantfunction F is correctly speciﬁed under a surrogate loss V ( · ) for the weighted method, if for any0 ≤ w ≤ 1 with P ( η ( X ) = w ) = 0 , any solution f (cid:63)w,V ( · ) to (11) satisﬁes { x : f (cid:63)w,V ( · ) ( x ) > 0 } ={ x : η ( x ) > w } up to a P X -null set.
Under this notion of correct model speciﬁcation, we can indeed show that the weighted method usingthe surrogate loss V ( · ) recovers the optimal ROC curve if the model space F is correctly speciﬁedunder V ( · ) .
Theorem 3 If F is correctly speciﬁed under a surrogate loss V ( · ) for the weighted method, then wehave thatγ W,V ( · ) ( F ) =
(cid:8) ( α, s ( α )) :
s ( α ) is non-differentiable at α, α ∈ (0 , 1 − P ( η ( X ) = 0) /p − ) (cid:9) ∪ (cid:8)
[Equation]
[Equation]
= 0 , α ∈ (0 , 1 − P
[Equation]
( X ) = 0) /p
[Equation]
{+) (cid:9) ∪ { ( FPR ( h (cid:63)w,V ( · ) ) , TPR ( h (cid:63)w,V ( · ) )) : w ∈ [0 , 1] and P ( η ( X ) = w ) > 0 } ∪ S (cid:48) 
+}
∪ S (cid:48) 1 , (13) where S (cid:48) 0 and S (cid:48) 1 satisﬁes
{+S (cid:48) 
+}
= (cid:26) { (0 , s (0)) } , if c (0) < 1 or P ( η ( X ) = 1) = 0 , ∅ , otherwise , (14) and S (cid:48) 1 = (cid:26) { (1 − P ( η ( X ) = 0) /p − , 1) } , if lim α → (1 − P (
[Equation]
( X )=0) /p − ) − c ( α ) > 0 or P ( η ( X ) = 0)
= 0 , ∅ , otherwise .
Theorem 3 connects the population ROC curve of the weighted method to the optimal ROC curvewhen H
= H a .
In Proposition S1 of the supplementary material, we establish that the set of ( α, s ( α ))with P ( η ( X ) =
c ( α )) > 0 and α > 0 corresponds to linear pieces on γ ( H a ) .
Combining this with(13) in Theorem 3, it follows that except for some interior points of these linear pieces the other partsover optimal ROC curve can be recovered by weighted method.
Therefore, roughly speaking, the setof points on the optimal ROC curve that can be recovered by the weighted method is the union ofnon-differentiable part and nonlinear part (up to some isolated points).
Although the above result seems to suggest that the weighted method may not recover all these linearpieces on the optimal ROC curve, it can be trivially modiﬁed to recover the entire optimal ROC curveby considering the pareto frontier of the convex hull of the ROC curve generated by weighted method.
This is established in the {+following corollary. Corollary 
+}
If F is correctly speciﬁed under a surrogate loss V ( · ) for the weighted method, then wehave that [Equation] where conv ( γ W,V ( · ) ( F )) denotes the closed convex hull of γ W,V ( · ) ( F ) . Next, we establish sufﬁcient conditions on V ( · ) under which F a is correctly speciﬁed under V ( · ) ,where F a consists of all measurable functions.
Interestingly, the requirement on V ( · ) coincides withthe notion of classiﬁcation-calibrated [Bartlett et al., 2006].
Deﬁnition 3 (Classiﬁcation-calibrated loss [Bartlett et al., 2006]) Let 0 ≤ η ≤ 1 and deﬁne
[Equation] [Equation]
(18) where V ( · ) is a surrogate loss function.
V ( · ) is said to be classiﬁcation-calibrated if H V ( η ) <
H − V ( η )for any η (cid:54) = 1 / 2 .
In Bartlett et al.
[2006], it is shown that using a classiﬁcation-calibrated loss V ( · ) ensures consistentestimation of the optimal misclassiﬁcation error, when F includes all measurable function (i.e.,
F = F a ).
Now we establish that the class of all measurable discriminant functions is correctlyspeciﬁed under any classiﬁcation-calibrated surrogate loss.
Proposition 2 The class of all measurable functions F a is correctly speciﬁed for the weighted method(i.e.
, in the sense of Deﬁnition 2) under the 0 / 1 loss, and any classiﬁcation-calibrated surrogate loss
V ( · ) .
Combining this with Theorem 3, it follows that if V ( · ) is classiﬁcation-calibrated and F = F a , thenthe weighted problem using the surrogate loss can recover the optimal ROC curve.
In other words,for weighted method using classiﬁcation-calibrated loss over F = F a generates consistent estimationof the optimal ROC curve.
Finally, we show that when the model space F is mis-speciﬁed under a surrogate loss V ( · ) , thepopulation ROC curve of the weighted method may miss some components of optimal ROC curve.
Moreover, these components cannot be recovered even after taking the convex hull transformation.
Theorem 4 Denote H =
{ Sign ( f ) | f ∈ F} .
In general, it is possible thatγ ( H ) \ γ (cid:63)W,V ( · ) ( F ) (cid:54) = ∅ and γ ( H ) \ PF ( conv ( γ W,V ( · ) ( F )))
(cid:54) = ∅ .
3.3 Cutoff method
There is another type of methods commonly used in practice called the “cutoff method”.
It ﬁrstproduces a discriminant function using some classiﬁcation method, and then generates a set ofclassiﬁers by thresholding a sequence of cutoffs to the discriminant function.
In order to study thepopulation analogue of the cutoff method, we assume that the discriminant function is generated froma given classiﬁcation method with class weight w ∈ [0 , 1] for negative instances:
[Equation]
20) where F is the set of discriminant functions.
We deﬁne the population ROC curve of the cutoffmethod over F to be
[Equation]
Next, we establish sufﬁcient conditions on F and V ( · ) to ensure that the cutoff method can recover theoptimal ROC curve.
Similar to the weighted method, we deﬁne a notion of correct model speciﬁcationfor the cutoff method.
Deﬁnition 4 (Correct model speciﬁcation for cutoff method)
We say that a set of discriminantfunction F is correctly speciﬁed under a surrogate loss function V ( · ) for the cutoff method for somew ∈ (0 , 1) , if any solution to (20) at w is a strictly increasing transformation to η ( x ) almost surely,that is,
[Equation]
surely , where M ( · ) is a strictly increasing function.
Under this notion of correct model speciﬁcation, we show that the cutoff method targets the optimal
ROC curve when η ( X ) is a continuous random variable.
This is established in the following theorem.
Theorem 5 Assume η ( X ) is a continuous random variable.
If F is correctly speciﬁed under asurrogate loss V ( · ) for the cutoff method for some w ∈ [0 , 1] , then we have that
[Equation]
In other words, the cutoff method using surrogate loss V ( · ) over F can recover the optimal ROCcurve.
For the cutoff method, we can show that the set of all measurable discriminant functions is correctlyspeciﬁed under any surrogate loss V ( · ) that is differentiable, strictly decreasing, and strictly convex.
Theorem 6 Assume η ( X ) is a continuous random variable.
Suppose that V ( · ) is a differentiable,strictly decreasing, proper and strictly convex loss.
Then the class of all measurable functions F a iscorrectly speciﬁed under V ( · ) for the cutoff method at any w ∈ (0 , 1) .
An important implication of the above two theorems is that, the cutoff method deﬁned in (21) canrecover the optimal ROC curve if F = F a and the surrogate loss V ( · ) is differentiable, strictlydecreasing and strictly convex.
Note that the requirement on the surrogate loss for the modelto be correctly speciﬁed for the cutoff method is more stringent than the classiﬁcation-calibratedrequirement for the weighted method (c.f., Lemma S7 in the supplementary material).
For a general model space F , the requirement of correct model speciﬁcation for the cutoff methodis also generally stronger than those needed for the weighted method.
This is because in general we would not know whether f (cid:63)w,V ( · ) ( x ) is an strictly increasing function of η ( x ) , even if
Sign( f (cid:63)w,V ( · ) ( x ))
=
Sign( η ( x ) − w ) and η ( X ) is a continuous random variable.
In our next theorem,we give an example where the population ROC curve generated by the cutoff method over F may bedominated by the optimal ROC curve over H =
{ Sign ( f − δ ) | f ∈ F , δ ∈ R } , suggesting that thecutoff method may not recover the optimal ROC curve over H in general.
Theorem 7 The population ROC curve of the cutoff method over certain model class F may bedominated by the optimal ROC curve over H = { Sign ( f − δ ) | f ∈ F , δ ∈ R } , in the sense thatthere exists a situation where for some ( u, v ) ∈ γ T,V ( · ) ( F , w ) , there exists ( u (cid:48) , v (cid:48) ) ∈ γ
( H ) such that( u (cid:48) , v (cid:48) ) (cid:31) ( u, v ) .  Simulations In this section, we examine the operating characteristics of the three methods analyzed in this article.
In our simulation, we investigate the effect of model mis-speciﬁcation by considering linear classiﬁersover two simulated data sets.
One dataset is generated under linear discriminant analysis (LDA)setting, which imitates the scenarios when model is correctly speciﬁed.
The other is based on aquadratic discriminant analysis (QDA) setting, which shows the effects of model mis-speciﬁcation.
Throughout, we consider using linear classiﬁcation methods in the experiments, including constrainedψ -learning, weighted SVM and its cutoff version.
We also consider kernel weighted SVM and itscutoff version.
This allows us to investigate the effect model mis-speciﬁcation on the performance of ROC curve estimation.
Example 1 (LDA setting).
This example considers a linear discriminant analysis model, where 369 P ( Y = 1) =
P ( Y = − 1) = 1 / 2 , and conditioned on Y , X | Y ∼ N ( Y µ , 4 I ) , where µ =(1 , 1) (cid:62) ∈ R 2 .
The sample size is n = 2000 .
Example 2 (QDA setting).
This example considers a quadratic discriminant analysis model,where P ( Y = 1) =
P ( Y = − 1) = 1 / 2 , and conditioned on Y , X | Y ∼ N ( Y µ , Σ Y ) , whereµ =
(1 , 1) (cid:62) ∈ R 2 , and [Equation]
The sample size is n = 2000 .
For both examples, we compare the average estimated ROC curve for the ﬁve aforementioned methodsover 100 replicates.
To generate the estimated ROC curve, we vary weight w and the constraint upperbound α from { i/ 500 | i = 0 , 1 , . . . , 500 } for the weighted method and the constrained method,respectively.
For the two cutoff methods, we choose w = 1 / 2 .
The results are summarized in Figure S1 of supplementary material.
The left panel of Figure S1 is for the LDA setting, where the modelis correctly speciﬁed for all methods.
In this case, all methods perform almost the same.
The rightpanel of Figure S1 is the for the QDA setting.
In this case, the model is mis-speciﬁed for linearmethods.
As expected, the two kernel methods perform the best.
Moreover, among linear methods,the constrained method performs the best, which corroborates with our theoretical ﬁndings.
Discussion
In this article, we deﬁne a general notion of optimal ROC curve, which we argue should be the theoretical target of any ROC curve estimation methods.
We analyze three popular ROC estimation methods and show that when the model space contains all measurable discriminant functions, the constrained method and the weighted method are both able to target at the optimal ROC curve under the 0 / 1 loss or some class of surrogate loss functions. The cutoff method however, requires a stronger set of conditions on the surrogate loss for it to be consistent.
When model mis-speciﬁcation is present, we show that only the constrained method under the 0 / 1 loss targets the optimal ROC curve over the used model space.
The two other methods may miss some FPR-TPR pairs on the optimal ROC curve. This suggests that the constrained method with a non-convex surrogate loss would be the preferred method when the model is clearly mis-speciﬁed.
However, for nonparametric classiﬁcation methods, the weighted method may be preferred with an appropriate choice of surrogate loss
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu-
tions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] We will release our codes later.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experimentsmultiple times)?
[No] We report the average performance of experiments results by multiplereplicates.
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs,internal cluster, or cloud provider)?
[No] Computational speed is not the main concern of oursimulations.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[N/A
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable informa-
[Table]
tion or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Table]
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) [Table]
approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on [Table]
participant compensation?
[N/A]
[Table]