On the consistent estimation of optimal Receiver Operating Characteristic (ROC) curve
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Under a standard binary classiﬁcation setting with possible model misspeciﬁcation,we study the problem of estimating general Receiver Operating Characteristic(ROC) curve, which is an arbitrary set of false positive rate (FPR) and true positiverate (TPR) pairs.
We formally introduce the notion of optimal ROC curve overa general model space.
It is argued that any ROC curve estimation methodsimplemented over the given model space should target the optimal ROC curve overthat space.
Three popular ROC curve estimation methods are then analyzed at thepopulation level (i.e., when there are inﬁnite number of samples) under both correctand incorrect model speciﬁcation.
Based on our analysis, they are all consistentwhen the surrogate loss function satisﬁes certain conditions and the given modelspace includes all measurable classiﬁers.
Interestingly, some of these conditionsare similar to those that are required to ensure classiﬁcation consistency.
When themodel space is incorrectly speciﬁed, however, we show that only one method leadsto consistent estimation of the ROC curve over the chosen model space.
We presentsome numerical results to demonstrate the effects of model misspeciﬁcation on theperformance of various methods in terms of their ROC curve estimates.
Introduction
One common goal in standard binary classiﬁcation task is to ﬁnd a classiﬁer with minimum misclassiﬁcation error, which is the average of the false negative rate (FNR) and false positive rate (FPR).
This implicitly assumes that the two types of errors are equally important.
In many real applications,however, these two types of errors may need to be treated unequally by assigning asymmetric lossesto them.
For example, in spam email detection, classifying non-spam as spam may incur a highercost than classifying spam into non-spam.
Existing methods to handle this problem can be roughlycategorized into three groups.
As a generalization to the empirical risk minimization framework[Vapnik, 1991], cost-sensitive learning [Elkan, 2001] assigns unequal class weights to the two types oferrors and seeks to ﬁnd a classiﬁer that minimizes the weighted error.
Neyman-Pearson classiﬁcation[Cannon et al., 2002; Scott and Nowak, 2005; Rigollet and Tong, 2011] is another type of method,which aims to minimize one type of error (FNR) subject to a upper bound constraint on the othertype of error (FPR).
We call these two types of methods weighted method and constrained method ,respectively.
Another popular approach [Fawcett, 2006], which we call cutoff methods , directlyconstructs a set of classiﬁers based on a discriminant function from some classiﬁer.
Speciﬁcally,the set of classiﬁers is obtained through changing the intercept of the given discriminant function.
Empirically, the cutoff method is computationally efﬁcient and can work well in some situations.
To study the performance of these methods at the population level, i.e., there are inﬁnite number ofsamples, we ﬁrst introduce a general notion of optimal ROC curve over an arbitrary model space.
Thisnew notion of optimal ROC curve parallels the notion of Bayes error for standard binary classiﬁcation when the model space is the set of all measurable classiﬁers.
Traditionally, ROC curve is typicallydeﬁned as the set of FPR-TPR pairs produced by the classiﬁers obtained from the cutoff method,where TPR denotes true positive rate and is deﬁned by TPR = 1 − FNR.
An optimal ROC curve[see, e.g., Van Trees, 2004] is deﬁned as the ROC curve generated by η ( x ) =
P ( Y = 1 | X = x ) ,where Y denotes the binary response and X denotes the feature vector.
This curve is optimal in thesense that no other ROC curves can lie above it.
Here we generalize the notion of optimal ROCcurve given a speciﬁc model space.
This is done by assigning a partial order to all possible FPR-TPRpairs that can be obtained over the given model space, and deﬁning the optimal ROC curve as thepareto frontier of the set of FPR-TPR pairs.
This allows us to deﬁne the optimal ROC curve under anarbitrary model space, and discuss the general case that model is possibly mis-speciﬁed.
We showthat the newly deﬁned optimal ROC curve coincides with the existing deﬁnition when the modelspace includes all possible measurable classiﬁers.
In terms of consistency, we ﬁnd that when the model space contains all measurable classiﬁers, allthree methods target the optimal ROC curve under 0 / 1 loss or some surrogate loss under a newnotion of correct model speciﬁcation.
Interestingly, in order for a model-loss pair to be correctlyspeciﬁed, a sufﬁcient condition required for the surrogate loss is related to the notion of classiﬁcationcalibration [Bartlett et al., 2006], which is a sufﬁcient condition required to achieve consistency inbinary classiﬁcation.
In this sense, our results generalize that of [Bartlett et al., 2006] for binaryclassiﬁcation to optimal ROC curve estimation.
When the model-loss pair is mis-speciﬁed, however,we show that only the constrained method targets the optimal ROC curve over the chosen modelspace.
In this sense, the other two methods may miss some optimal FPR-TPR pairs on the optimal
ROC curve.
Importantly, this suggests that the constrained method may be preferable to the two othermethods in many practical applications where the model space considered does not include the truemodel.
To the best of our knowledge, our work is the ﬁrst that formally investigate theoretical conditionsunder which the commonly used methods could recover the optimal ROC curve in a statisticaldecision theoretical framework.
Moreover, our proposed deﬁnition of optimal ROC curve alsogeneralizes the conventional deﬁnition, relaxing some unnecessary distributional assumptions onthe data.
For ROC curve estimation, existing works are either empirical or mostly focus on how togenerate a single FPR-TPR pair on the optimal ROC curve using either the weighted method or theconstrained method.
For example, Cannon et al.
[2002] investigated the empirical version of theconstrained method, and provided a probabilistic guarantee on simultaneous control of both FPR and
FNR.
Extensions to structural risk minimization were later considered in Scott and Nowak [2005].
However, they did not address the issues on how to better estimate the whole ROC curve.
Scott [2007]proposed two families of performance measures to evaluate classiﬁers produced by the constrainedmethod.
Interestingly, they also showed that the ROC curve is concave under some assumptions onthe data generating distributions, part of which are special cases of our theory (see Theorem 2).
Morerecently, Rigollet and Tong [2011] focuses on how to obtain a classiﬁer that can control its FPR atthe population level.
This is done by using a convex surrogate loss and considering a slightly morestringent upper bound than the desired level for the FPR.
However, their method may not target thebest trade-off between FPR and TPR, which is partly attributed to the use of a convex surrogate lossand possible model mis-speciﬁcation.
Lastly, Bach et al.
[2006] proposed to estimate the ROC curveby computing a convex envelope of ROC curves generated by the cutoff method when varying theclass weights.
They demonstrate that this will usually outperform the ROC curve produced by thecutoff method alone, which is partly consistent with our theory that the weighted and cutoff methodmay be sub-optimal when the model is mis-speciﬁed.
Moreover, our deﬁnition of the optimal ROC curve is also new and it extends the old deﬁnitionconsidered by Van Trees [2004]; Scott [2007] in two different ways.
First, the optimal ROC curvein [Van Trees, 2004; Scott, 2007] requires some continuity assumptions on the data generatingdistribution.
However, our deﬁnition is quite general and does not make any assumptions on the datagenerating distribution.
In fact, we show that our deﬁnition coincides with the existing deﬁnitionif the continuity assumption is satisﬁed.
Second, our deﬁnition allows model mis-speciﬁcation,which reveals some interesting differences among the commonly used methods when model misspeciﬁcation may be present.
Another important contribution is that we formally prove the continuityand concavity of the optimal ROC curve and present some other important properties of the optimal
ROC curve, which extends results informally stated in [Van Trees, 2004; Scott, 2007].
The rest of the article is structured as follows.
We deﬁne the notion of optimal ROC curve overa general model space, and derive the optimal ROC curve over the entire population analyticallyin Section 2.
Section 3 analyzes the population analogue of three ROC estimation methods whenmodel is correctly speciﬁed and mis-speciﬁed.
We present some numerical examples to illustratethe impact the mis-speciﬁcation on the performance of three methods in Section 4, and we end withsome discussions in Section 5.
All proofs are included in appendix.
Optimal ROC curve over a general model space
In this section, we introduce the notion of optimal ROC curve over a general model space.
The notionof optimal ROC curve is parallel to Bayes risk in standard binary classiﬁcation.
We therefore arguethat the optimal ROC curve should be the theoretical target for ROC curve estimation, just like Bayeserror in standard binary classiﬁcation.
Before proceeding, we ﬁrst lay out the basic problem setup, some notations, and assumptions to beused throughout the article.
Consider a feature-response pair ( X, Y ) with joint distribution P , where
X ∈ R p is the feature vector and Y ∈ {− 1 , 1 } is the corresponding binary response.
Throughout,we make the assumption that the marginal distribution X , denoted as P X , is continuous, and themarginal distribution of Y satisﬁes p + = P
( Y = 1) ∈ (0 , 1) and p − =
P ( Y = − 1) ∈ (0 , 1) .
Weuse h :
R p
→
{− 1 , 1 } to denote a classiﬁer that maps feature vectors to their responses.
For manymethods, a classiﬁer is induced by the sign of a so-called discriminant function .
Throughout, we use
H to denote set of classiﬁers, and F to denote set of discriminant functions.
Many different metrics can be used to assess the performance of a classiﬁer, of which misclassiﬁcationerror rate P ( Y (cid:54)
=
h ( X )) is perhaps the most commonly used and well-studied in the literature[Zhang et al., 2004; Bartlett et al., 2006].
Classiﬁcation methods are often designed to minimizethe misclassiﬁcation error, which is the weighted average of two types of error rate, namely, falsepositive rate (FPR): FPR ( h ) =
P ( h ( X ) = 1 | Y = − 1) and false negative rate (FNR): FNR ( h ) = {+
+}
P ( h ( X ) =
− 1 | Y = 1) .
In this article, we focus on the problem of minimizing these two typesof error rate simultaneously.
This is essentially equivalent to the estimation of optimal ROC curve,which plots the true positive rate (TPR) against the false positive rate for a set of classiﬁers, wherethe true positive rate is deﬁned to be TPR ( h )
= 1 − FNR
( h ) .
In standard binary classiﬁcation, the optimal classiﬁer that minimizes the misclassiﬁcation error isoften referred to as the Bayes classiﬁer.
For our problem, we have two criterion to optimize.
Byborrowing ideas from multi-objective optimization literature [Miettinen, 2012], we deﬁne the setof optimal classiﬁers as those that can not be dominated by any other classiﬁers, and the optimal
ROC curve as the set of FPR-TPR pairs that can not be dominated by any other FPR-TPR pairs.
Here classiﬁer h dominates classiﬁer h (cid:48) , denoted as h (cid:31) h (cid:48) , means that TPR ( h ) ≥ TPR ( h (cid:48) ) ,
FPR ( h ) ≤ FPR ( h (cid:48) ) and ( FPR ( h ) , TPR ( h )) (cid:54) =
( FPR ( h (cid:48) ) , TPR ( h (cid:48) )) .
In this sense, a classiﬁer h is anoptimal classiﬁer if it can not be dominated by any other classiﬁers.
Similarly, for any two FPR-TPRpairs ( u, v ) and ( u (cid:48) , v (cid:48) ) , we say that ( u, v ) dominates ( u (cid:48) , v (cid:48) ) , denoted as ( u, v ) (cid:31) ( u (cid:48) , v (cid:48) ) , if u ≤ u (cid:48) ,v ≥ v (cid:48) and ( u, v ) (cid:54) =
( u (cid:48) , v (cid:48) ) .
Given this partial ordering, we denote by PF ( A ) :
= { ( u, v ) ∈
A |( u (cid:48) , v (cid:48) )
(cid:7) ( u, v ) for any ( u (cid:48) , v (cid:48) )
∈ A } the pareto frontier of a compact set A ⊆ R 2 .
Therefore, theset of optimal classiﬁers constitutes the pareto frontier of the set of all measurable classiﬁers, whilethe optimal ROC curve is the pareto frontier of the set of all possible FPR-TPR pairs.
Formally, we deﬁne the optimal ROC curve over an arbitrary set of classiﬁers H , which allowspossible model mis-speciﬁcation .
Deﬁnition 1 (Optimal ROC curve over model space H )
Given a set of classiﬁers H , denote by
S ( H ) =
{ ( FPR ( h ) , TPR ( h ))
| h ∈ H} the set of all FPR-TPR pairs generated by classiﬁers in H .
We call the pareto frontier of S ( H ) , denoted as γ ( H ) =
PF ( S ( H )) , the optimal receiver operatingcharacteristic (ROC) curve over H .
When H =
H a , where H a is the class of all measurableclassiﬁers, we simply call γ ( H a ) the optimal ROC curve.
Throughout this article, we assume that S ( H ) is a closed set so that PF ( S ( H )) is well-deﬁned.
Aswe can see from Deﬁnition 1, classiﬁers corresponding to any point on the optimal ROC curve cannotbe dominated by any other classiﬁer.
Note that this is also similar to the notion of admissibilityfor general decision rules in statistical decision theory.
The above deﬁnition of optimal ROC curvediffers from the existing deﬁnition by Scott [2007] in that it allows for model mis-speciﬁcation.
When the model is correctly speciﬁed, the optimal ROC curve parallels the notion of Bayes error rate for regular binary classiﬁcation, while the corresponding classiﬁers parallels the notion of Bayesclassiﬁer.
It is well-known that Bayes error rate is 1 / 2 − E | η ( X ) − 1 / 2 | [Devroye et al., 2013], whereη ( x ) =
P ( Y = 1 | X = x ) is the regression function.
In parallel, we derive the optimal ROC curveover the class of all measurable classiﬁers H a .
Compared with some existing deﬁnitions of optimal
ROC curve under correct model speciﬁcation, our deﬁnition is more general, requiring less conditionson the distribution of ( X, Y )
[see, e.g., Scott, 2007].
Speciﬁcally, our deﬁnition of optimal ROCcurve only requires that X is a continuous random variable, while Scott [2007] requires that both Xand η ( X ) are continuous random variables.
The following theorem derives an analytic form for the optimal ROC curve γ ( H a ) deﬁned in
Deﬁnition 1.
Theorem 1 Let η ( x ) =
P ( Y = 1 | X = x ) , p (cid:63) =
inf { p : P ( η ( X ) ≤ p ) = 1 } , and let I ( · ) denotethe indicator function.
We have that the optimal ROC curve can be represented as
[Equation]
where
[Equation]
with c ( α ) =
inf { c ∈ [0 , p (cid:63) ] : E (1 − η ( X ))
( I ( η ( X ) > c ) − α ) ≤ 0 } .
Moreover, s ( α ) is continuous,strictly increasing, and concave over its domain [0 , 1 − P ( η ( X ) = 0)
/p − ] .
Some remarks are in order.
First, we can see that the optimal ROC curve is a continuous, strictly increasing, and concave curve deﬁned by the function s ( α ) over the interval [0 , 1 − P ( η ( X ) = 0) /p − ] .
Second, existing deﬁnition of the optimal ROC curve typically relies on the likelihood ratio test[see, e.g., McIntosh and Pepe, 2002; Scott, 2007].
More speciﬁcally, it assumes that X has classconditional density function f
+ ( x ) and f − ( x ) conditional on Y = 1 and Y = − 1 , and considers aclass of classiﬁers indexed by λ ∈ [0 , ∞ ] : { h λ ( x ) =
Sign ( f + ( x ) /f − ( x ) − λ ) , λ ∈ [0 , ∞ ] } .
Then,under the assumption that
[Equation]
for any λ ∈ [0 , ∞ ] , Scott [2007] deﬁned the optimal ROC curve as the set of FPR-TPR pairs generatedby the class of classiﬁers { h λ ( x ); λ ∈ [0 , ∞ ] } .
In other words, the existing deﬁnition of the optimal
ROC curve can be expressed as
[Equation]
Our deﬁnition of optimal ROC curve (c.f. Deﬁnition 1), however, does not require the likelihoodratio assumption in (3), and hence in this sense is more general than the existing deﬁnition.
Indeed,assumption(3) may be violated in some common situations, for example, the two class-conditionaldensities f + ( x ) and f − ( x ) have different supports.
Moreover, under the condition (3), we can showthat our proposed deﬁnition coincides with the existing deﬁnition (4).
Furthermore, the optimal ROCcurve is the FPR-TPR pairs generated by the set of classiﬁers { Sign( η ( X ) − c ) | c ∈ [0 , 1] } .
Wesummarize these results in Corollary S1 of the Appendix.
Lastly, we remark that our proposed notion of the optimal ROC curve (c.f. Deﬁnition 1) also allowsmodel misspeciﬁcation.
This allows us to study the statistical consistency of special classes ofclassiﬁcation method such as linear classiﬁers.
We will demonstrate in later sections that modelspeciﬁcation has a meaningful impact on the consistency of various popular classiﬁcation methodsfor ROC curve estimation.
ROC curve estimation methods
In this section, we review three ROC curve estimation methods that are widely used in practice.
We then investigate, from a statistical decision theoretical point of view, whether these methods aretargeting the optimal ROC curve deﬁned in the previous section.
In particular, we shall analyze theirpopulation analogue when the model is mis-speciﬁed, that is, the optimal classiﬁer is not included inthe set of classiﬁers considered in the model.
3.1 Neyman-Pearson classiﬁcation
Neyman-Pearson classiﬁcation [Scott and Nowak, 2005; Rigollet and Tong, 2011], or the constrainedmethod, minimizes one type of error subject to an upper bound constraint on the other type of error.
The population analogue of Neyman-Pearson classiﬁcation is equivalent to ﬁnding a classiﬁer thatmaximizes the true positive rate subject to false positive rate constraint, that is,maximize h ∈H
TPR ( h ) subject to FPR ( h ) ≤ α , (5) where H is a set of classiﬁers.
In this subsection, we will show that under mild conditions the ROCcurve generated from the constrained method by varying α ∈ [0 , 1] can recover the optimal ROCcurve over general model space H .
Before proceeding, we call the ROC curve generated by the constrained method, deﬁned by
[Equation]
, the population ROC curve of the constrained method, where h (cid:63)α, H is a solution to (5).
Note thatthe population ROC curve is essentially the set of FPR-TPR pairs generated by the solutions to theconstrained method (5) after varying α ∈ [0 , 1] .
We ﬁrst consider the case H
= H a , where the model space H a is the set of all measurable classiﬁers.
In this case, we can derive the analytic solution to the constrained method (5), which then allows usto derive the population ROC curve γ C ( H a ) generated by the constrained method.
Theorem 2 Denote η ( x ) =
P ( Y = 1 | X = x ) and p (cid:63) =
inf { p : P ( η ( X ) ≤ p ) = 1 } .
Let H
=
H abe the set of all measurable classiﬁers.
For any solution h (cid:63)α ( x ) of (5) , denote by R (cid:63)α = { x : h (cid:63)α ( x ) =1 } .
Then, up to a P X -null set,
[Equation]
where c ( α ) = inf { c ∈ [0 , p (cid:63) ] : E (1 − η ( X ))
( I ( η ( X ) > c ) − α ) ≤ 0 } with c ( α ) ∈ [0 , 1) and
N α ⊆ { x : η ( x ) =
c ( α ) } with
[Equation]
Moreover, we have that optimal ROC curve γ ( H a ) satisﬁes
[Equation]
This theorem provides a complete characterization of the population ROC curve of the constrainedmethod.
Combining (8) with Theorem 1, we can see that the constrained method can recover theoptimal ROC curve in the sense that it produces a curve that contains the optimal ROC curve.
Using this, we can further show that the pareto frontier of the population ROC curve, deﬁned byγ (cid:63) C ( H a ) =
PF ( cl ( γ C ( H a ))) , is identical to the optimal ROC curve. Here we use cl ( A ) to denote theclosure of set A . We summarize this result as below.
Corollary [-
-]1 We have that
[Equation]
Corollary 1 suggests that the constrained method over H a indeed targets the optimal ROC curveγ ( H a ) .
Essentially, by our proof of Corollary 1, the discrepancy between γ C ( H a ) and the optimal
ROC curve γ ( H a ) occurs only if TPR = 1 .
Moreover, by our proof if P ( η ( X ) = 0)
= 0 thenwe have that γ C ( H a ) =
γ ( H a ) .
Notably, Scott [2007] also derived the ROC curve generated bythe constrained method γ C ( H a ) and proved that it coincides with the optimal ROC curve theydeﬁned under likelihood ratio condition (3).
Here, we show that the ROC curve generated by theconstrained method coincides with our deﬁned optimal ROC curve (c.f. deﬁnition 1) by only requiring 224 P ( η ( X ) = 0)
= 0 .
Note that for a general model space H , it is no longer possible to derive the population ROC curveanalytically.
However, relations like (9) continue to hold.
That is, the pareto frontier of the population ROC curve for the constrained method, deﬁned by γ (cid:63)C ( H ) =
PF ( cl ( γ C ( H ))) , coincides with theoptimal ROC curve over H .
Proposition 1 Suppose that S ( H ) =
{ ( FPR ( h ) , TPR ( h ))
| h ∈ H} is a closed set.
We have thatγ (cid:63)C ( H ) =
γ ( H ) ⊆
γ C ( H ) .
Proposition 1 shows that any point on the optimal ROC curve over a general model space H can berecovered by solutions to the constrained method (5) for some α ∈ [0 , 1] .
Later, we shall show thatthis property is unique to the constrained method for a general model space H .
So far all results consider the 0 / 1 loss.
In practice, a surrogate loss is often used to replace the 0 / 1loss, and a discriminant function is often used as the decision variable in the optimization problem.
Here we propose to use the so-called psi loss ψ ( x ) = min(1 , max(0 , 1 − x ))
[Shen et al.
, 2003],which could approximate the 0 / 1 loss up to arbitrary precision.
The rationale here is that a continuousloss that is close to the 0 / 1 loss would enjoy similar nice theoretical properties of the 0 / 1 loss.
Asa ﬁnal note, we do not discuss convex surrogate loss for the constrained method, partly because ifconvex surrogate loss and convex set of discriminant function are used, the optimization problemassociated with the constrained method is convex, and thus is equivalent to a weighted problem,which we will study subsequently in the next subsection.
3.2 Weighted method
Another popular method to estimate the ROC curve, which we call the weighted method, considers aset of classiﬁers obtained by assigning different class weights to the two types of errors.
In practice,instead of directly optimizing the “weighted error rate”, some continuous surrogate loss (oftenconvex) is typically used to replace the non-continuous 0 / 1 loss for ease of computation, whichcould introduce additional “model bias” or model mis-speciﬁcation.
This phenomenon has beenextensively studied for standard binary classiﬁcation task [see, e.g., Zhang et al., 2004; Bartlett et al.
Roughly speaking, the surrogate loss function needs to satisfy certain condition to ensurethe consistency of the procedure (in terms of Bayes error).
In Bartlett et al.
[2006], the surrogateloss satisfying this condition is called to be classiﬁcation-calibrated [Bartlett et al., 2006], and theyfurther showed that procedures that are consistent using the 0 / 1 loss, are also consistent using anyclassiﬁcation-calibrated losses.
Here we shall present some parallel results for ROC curve estimation.
More speciﬁcally, we considerthe population analogue of the weighted method under a surrogate loss V ( · ) over a set of discriminantfunctions F :
[Equation]
We denote by f (cid:63)w,V ( · ) a solution to (10), which may not necessarily be unique.
Given the optimaldiscriminant function f (cid:63)w,V ( · ) , we deﬁne the associated optimal classiﬁer by h (cid:63)w,V ( · ) =
Sign( f (cid:63)w,V ( · ) ) ,where Sign ( z ) =
I ( z ≤ 0) − 1 denotes the sign function.
We aim to ﬁnd conditions for the surrogateloss and the class of discriminant function, under which the population ROC curve of the weightedmethod using a surrogate loss V ( · ) over a set of discriminant functions F , deﬁned as
[Equation]
can recover the optimal ROC curve over H =
{ Sign( f ) | f ∈ F} .
We ﬁrst present a generalcondition on the class of discriminant functions F and the surrogate loss function V ( · ) to ensureconsistent estimation of the optimal ROC curve.
Deﬁnition 2 (Correct model speciﬁcation for weighted method)
We say that a set of discriminantfunction F is correctly speciﬁed under a surrogate loss V ( · ) for the weighted method, if for any0 ≤ w ≤ 1 with P ( η ( X ) =
w ) = 0 , any solution f (cid:63)w,V ( · ) to (10) satisﬁes { x : f (cid:63)w,V ( · ) ( x ) > 0 }
={ x : η ( x ) > w } up to a P X -null set.
Under this notion of correct model speciﬁcation, we can indeed show that the weighted method usingthe surrogate loss V ( · ) recovers the optimal ROC curve if the model space F is correctly speciﬁedunder V ( · ) .
Theorem 3 If F is correctly speciﬁed under a surrogate loss V ( · ) for the weighted method, then wehave that
[Equation]
is non-differentiable at α
[Equation]
α ∈
[Equation]
P
[Equation]
[-) (cid:9) ∪ { ( FPR ( h (cid:63)w,V ( · ) ) , TPR ( h (cid:63)w,V ( · ) )) : w ∈ [0 , 1] and P ( η ( X ) = w ) > 0 } ∪ S (cid:48) 
-]
where S (cid:48) 0 and S (cid:48) 1 satisﬁes [Equation] [Equation] 0 , s (
[-S (cid:48) 
-][Equation]
} , if c [Equation] [Equation] < 1 or P
[Equation] η
[Equation]
X ) = 1) = 0 , ∅ , otherwise , (13) and [Equation] p [Equation]
[Equation]
[Equation] ∅ , otherwise .
Theorem 3 connects the population ROC curve of the weighted method to the optimal ROC curvewhen H
= H a .
In Proposition S1 of the supplementary material, we establish that the set of ( α, s ( α ))with P ( η ( X ) =
c ( α )) > 0 and α > 0 corresponds to linear pieces on γ ( H a ) .
Combining this with(12) in Theorem 3, it follows that except for some interior points of these linear pieces the other partsover optimal ROC curve can be recovered by weighted method.
Therefore, roughly speaking, the setof points on the optimal ROC curve that can be recovered by the weighted method is the union ofnon-differentiable part and nonlinear part (up to some isolated points).
Although the above result seems to suggest that the weighted method may not recover all these linearpieces on the optimal ROC curve, it can be trivially modiﬁed to recover the entire optimal ROC curveby considering the pareto frontier of the convex hull of the ROC curve generated by weighted method.
This is established in Corollary [-
-]S2 of the Appendix.
Next, we establish sufﬁcient conditions on V ( · ) under which F a is correctly speciﬁed under V ( · ) ,where F a consists of all measurable functions.
Interestingly, the requirement on V ( · ) coincides withthe notion of classiﬁcation-calibrated [Bartlett et al., 2006].
Deﬁnition 3 (Classiﬁcation-calibrated loss [Bartlett et al., 2006]) Let 0 ≤ η ≤ 1 and deﬁne
[Equation] [Equation]
(16) where V ( · ) is a surrogate loss function.
V ( · ) is said to be classiﬁcation-calibrated if H V ( η ) <
H − V ( η )for any η (cid:54) = 1 / 2 .
In Bartlett et al.
[2006], it is shown that using a classiﬁcation-calibrated loss V ( · ) ensures consistentestimation of the optimal misclassiﬁcation error, when F includes all measurable function (i.e.,
F = F a ).
Now we establish that the class of all measurable discriminant functions is correctlyspeciﬁed under any classiﬁcation-calibrated surrogate loss.
Proposition 2 The class of all measurable functions F a is correctly speciﬁed for the weighted method(i.e.
, in the sense of Deﬁnition 2) under the 0 / 1 loss, and any classiﬁcation-calibrated surrogate loss
V ( · ) .
Combining this with Theorem 3, it follows that if V ( · ) is classiﬁcation-calibrated and F = F a , thenthe weighted problem using the surrogate loss can recover the optimal ROC curve.
In other words,for weighted method using classiﬁcation-calibrated loss over F = F a generates consistent estimationof the optimal ROC curve.
Finally, we give an example in Theorem S1 of the Appendix to show that in contrast to the constrainedmethod (c.f. Proposition 1), the population ROC curve of the weighted method may miss somecomponents of optimal ROC curve when the model space F is mis-speciﬁed under a surrogateloss V ( · ) .
Importantly, these components cannot be recovered even after taking the convex hulltransformation.
3.3 Cutoff method
There is another type of methods commonly used in practice called the “cutoff method”.
It ﬁrstproduces a discriminant function using some classiﬁcation method, and then generates a set ofclassiﬁers by thresholding a sequence of cutoffs to the discriminant function.
In order to study thepopulation analogue of the cutoff method, we assume that the discriminant function is generated froma given classiﬁcation method with class weight w ∈
[0 , 1] for negative instances:
[Equation]
where F is the set of discriminant functions.
We deﬁne the population ROC curve of the cutoffmethod over F to be
[Equation]
Next, we establish sufﬁcient conditions on F and V ( · ) to ensure that the cutoff method can recover theoptimal ROC curve.
Similar to the weighted method, we deﬁne a notion of correct model speciﬁcationfor the cutoff method.
Deﬁnition 4 (Correct model speciﬁcation for cutoff method)
We say that a set of discriminantfunction F is correctly speciﬁed under a surrogate loss function V ( · ) for the cutoff method for somew ∈ (0 , 1) , if any solution to (17) at w is a strictly increasing transformation to η ( x ) almost surely,that is
,f (cid:63)w,V ( · ) ( X ) =
M ( η ( X )) almost surely , where M ( · ) is a strictly increasing function.
Under this notion of correct model speciﬁcation, we show that the cutoff method targets the optimal
ROC curve when η ( X ) is a continuous random variable.
This is established in the following theorem.
Theorem 4 Assume η ( X ) is a continuous random variable.
If F is correctly speciﬁed under asurrogate loss V ( · ) for the cutoff method for some w ∈ [0 , 1] , then we have that
[Equation]
For the cutoff method, we can show that the set of all measurable discriminant functions is correctlyspeciﬁed under any surrogate loss V ( · ) that is differentiable, strictly decreasing, and strictly convex.
Theorem 5 Assume η ( X ) is a continuous random variable.
Suppose that V ( · ) is a differentiable,strictly decreasing, proper and strictly convex loss.
Then the class of all measurable functions F a iscorrectly speciﬁed under V ( · ) for the cutoff method at any w ∈ (0 , 1) .
An important implication of the above two theorems is that, the cutoff method deﬁned in (18) canrecover the optimal ROC curve if F = F a and the surrogate loss V ( · ) is differentiable, strictlydecreasing and strictly convex.
Note that the requirement on the surrogate loss for the modelto be correctly speciﬁed for the cutoff method is more stringent than the classiﬁcation-calibratedrequirement for the weighted method (c.f., Lemma S7 in the supplementary material).
For a general model space F , the requirement of correct model speciﬁcation for the cutoff methodis also generally stronger than those needed for the weighted method.
This is because in general we would not know whether f (cid:63)w,V ( · ) ( x ) is an strictly increasing function of η ( x ) ,
even if
Sign( f (cid:63)w,V ( · ) ( x ))
=
Sign( η ( x ) − w ) and η ( X ) is a continuous random variable.
In Theorem S2 ofthe Appendix, we give an example to show that in contrast to the constrained method (c.f. Proposition1), the population ROC curve generated by the cutoff method over F may be dominated by theoptimal ROC curve over H =
{ Sign ( f − δ ) | f ∈ F , δ ∈ R } , suggesting that the cutoff method maynot recover the optimal ROC curve over H in general.
Simulations and real data analysis
In this section, we examine the operating characteristics of the three methods analyzed in thisarticle with both simulation studies and a real data example.
In our simulation, we investigate the effect of model mis-speciﬁcation by considering linear classiﬁers over two simulated data sets.
Onedataset is generated under linear discriminant analysis (LDA) setting, which imitates the scenarioswhen model is correctly speciﬁed.
The other is based on a quadratic discriminant analysis (QDA)setting, which shows the effects of model mis-speciﬁcation.
We also investigate the impact of modelmis-speciﬁcation by considering a bank marketing data set [Moro et al., 2014].
Throughout, weconsider using linear classiﬁcation methods in the experiments, including constrained ψ -learning,weighted SVM and its cutoff version.
We also consider kernel weighted SVM and its cutoff version.
This allows us to investigate the effect model mis-speciﬁcation on the performance of ROC curveestimation.
Example 1 (LDA setting).
This example considers a linear discriminant analysis model, where 361 P ( Y = 1) =
P ( Y = − 1) = 1 / 2 , and conditioned on Y , X | Y ∼ N ( Y µ , 4 I ) , where µ =(1 , 1) (cid:62) ∈ R 2 .
The sample size is n = 2000 .
Example 2 (QDA setting).
This example considers a quadratic discriminant analysis model,where P ( Y = 1) =
P ( Y = − 1) = 1 / 2 , and conditioned on Y , X | Y ∼ N ( Y µ , Σ Y ) , whereµ =
(1 , 1) (cid:62) ∈ R 2 , andΣ + = (cid:18) 85(cid:19) , Σ − = (cid:18)−− (cid:19) .
The sample size is n = 2000 .
Example 3 (Real data example).
This example considers a bank marketing dataset [Moro et al.
,2014], which records the direct marketing campaigns of a Portuguese banking institution andare available at https://archive.ics.uci.edu/ml/datasets/Bank+Marketing . This realdataset consists of 4521 examples with 17 features and 1 binary response variable. We visualize the scatter plots of two simulation examples in Figure S1. For two simulation examples,we compare the average estimated ROC curve for the ﬁve aforementioned methods over 100 replicates;while for the real data example, we present the estimated ROC curves directly.
For all three examples,to generate the estimated ROC curve we vary weight w and the constraint upper bound α from{ i/ 500 | i = 0 , 1 , . . . , 500 } for the weighted method and the constrained method, respectively.
Forthe two cutoff methods, we choose w = 1 / 2 .
The results are summarized in Figure S2 and Figure S3of supplementary material.
The left panel of Figure S2 is for the simulations under LDA setting, where the model is correctlyspeciﬁed for all methods.
In this case, all methods perform almost the same.
The right panel of Figure S2 is the for the simulations under QDA setting.
In this case, the model is mis-speciﬁed for linearmethods.
As expected, the two kernel methods perform the best.
Moreover, among linear methods,the constrained method performs the best, which corroborates with our theoretical ﬁndings thatconstrained method performs best under possible model mis-speciﬁcation.
For our real data example,we can see in Figure S3 that among all these linear method the constrained method performs best,but is no better than the weighted kernel SVM.
These results again corroborate with our theoreticalanalysis on the goodness of using constrained method when model is mis-speciﬁed.
Discussion
In this article, we deﬁne a general notion of optimal ROC curve, which we argue should be thetheoretical target of any ROC curve estimation methods.
We analyze three popular ROC estimationmethods, and show that when the model space contains all measurable discriminant functions allthree methods target the optimal ROC curve, although the cutoff method requires a stronger set ofconditions on the surrogate loss for it to be consistent.
When model mis-speciﬁcation is present,we show that only the constrained method under the 0 / 1 loss targets the optimal ROC curve overthe used model space.
This together with our simulations and real data studies suggests that theconstrained method with a non-convex surrogate loss (e.g. ψ loss) would be the preferred methodwhen the model is possibly mis-speciﬁed. Although we have observed good empirical performancesusing the constrained method with ψ loss, we have not investigated its theoretical guarantees in thisarticle.
Further theoretical research on non-convex surrogate loss to achieve guaranteed performancemay be necessary.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu-
tions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[No] We will release our codes later.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No] We report the average performance of experiments results by multiple replicates.
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs,internal cluster, or cloud provider)?
[No] Computational speed is not the main concern of oursimulations.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
informa-
tion or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on
participant compensation?
[N/A]