Estimating the Arc Length of the Optimal ROC Curve and Lower Bounding the Maximal AUC
Abstract
In this paper, we show the arc length of the optimal ROC curve is an f -divergence.
By leveraging this result, we express the arc length using a variational objective and estimate it accurately using positive and negative samples.
We show this estimator has a non-parametric convergence rate O p ( n − β/ 4 ) ( β ∈ (0 , 1] depends on the smoothness).
Using the same technique, we show the surface area between the optimal ROC curve and the diagonal can be expressed via a similar variational objective.
These new insights lead to a novel classification procedure that maximizes an approximate lower bound of the maximal AUC.
Experiments on CIFARdatasets show the proposed two-step procedure achieves good AUC performance in imbalanced binary classification tasks.
Introduction
The study of Receiver operating characteristic (ROC) curves has a long history in medicine [24], psychology [13] and radiology [14, 11].
In machine learning, ROC curves have been primarily used to analyze the performance of different classification algorithms [6, 7].
Indeed, the Area Under the
Curve (AUC) encodes a classifier’s ranking accuracy, making it a preferable performance metric for imbalanced class classification [6, 2].
In recent years, ROC curves have also been used in comparing two distributions and achieved promising results.
Examples include analyzing the mode collapsing issue of Generative Adversarial nets (GAN) [23], and diagnosing the performance of an amortized
Markov Chain Monte Carlo [16].
In applications that require computing statistical discrepancy between distributions (e.g. GAN [12] or Variational Inference (VI) [1]), f -divergences are widely used discrepancy measures.
The family of f -divergences includes Kullback-Leibler divergence [21] and Total Variation distance.
It has been shown that f -divergences, generally, can be expressed via variational objectives and efficiently approximated from empirical samples [27, 28].
Since the ROC curves are used as performance metrics in many two sample applications, are they in any way related to f -divergences?
For example, can AUC be an f -divergence between positive and negative data distributions given some classification score function?
An earlier investigation proves that the answer is no when the score function is the likelihood ratio [29].
Nonetheless, this result inspired us to look for f -divergence from other geometries of the ROC curve.
In this paper, we show that, when using the likelihood ratio score, a novel f -divergence arises from the arc length of the corresponding ROC curve.
By leveraging this result, we can express the arc length using a variational objective and approximate it using only samples from two distributions.
[Figure]
We show this arc length estimator is also a consistent estimator to the arctangent of likelihood ratio and has a non-parametric convergence rate O p ( n − β/ 4 ) , where β ∈ (0 , 1] depends on the smoothness of the true arctangent likelihood ratio.
Moreover, by parameterizing the ROC curves of positive and negative mixtures distributions, the surface area between the optimal ROC curve and the diagonal can be expressed via a similar variational objective.
With the help of our arctangent ratio estimator, we can approximately maximize a lower bound to this surface area.
We point out the similarity between this lower bound maximization and the classic AUC maximization [2].
We show our approximated optimal score achieves comparable performance to a state of the art AUC maximizer in an imbalanced classification problem on CIFAR-10 dataset.
Background 2.1 An Illustrative Example
ROC curves are frequently used to visualize binary classification performance, and we often rely on the Area Under the Curve (AUC) as a numerical metric for selecting a good classifier.
In this section, we highlight an often overlooked ROC geometry: arc length.
We illustrate its potential as a good discrepancy measure between positive and negative data distributions.
Let us consider the one-dimensional distributions and the ROC curves in Figure 1.
In this example,
ROC curves are generated using the identity score function t ( x ) = x .
In both (a) and (b), p + and p − are quite different, and thus the discrepancies between positive and negative distributions should be large in both cases.
In (c), the densities p + and p − are the same and consequently their discrepancy should be smaller than that of both (a) and (b).
Further, notice that both (a) and (b) have long ROC curves ( ≈ 2 ) while (c) has a shorter one ( ≈ √ 2 ).
This example suggests that the more similar p + and p − are, the shorter the ROC curve is.
However, we can see that t ( x ) = x is a special choice: If t ( x ) = 0 , the arc length will not reflect any discrepancy between data distributions at all.
This observation inspires the following questions: Why is the arc length in this example good at telling the differences between two data distributions?
Are there other score functions whose ROC arc lengths are also good discrepancy measures?
What are the practical applications of studying the arc length of ROC?
In the following sections, we study the arc length of a ROC curve under a probabilistic framework and provide answers to these questions.
2.2 ROC Curve in a Probabilistic Setting
Suppose we have positive and negative datasets X
+ :=
{ x + i } n + i =1 and X − := { x − i } n − i =1 drawn from two distributions P + and P − respectively.
These distributions have respective probability density functions p + ( x ) and p − ( x ) that are both defined on the domain X ⊆ R d .
A classification score function (score function for short) takes a sample x as input and outputs a real-valued score.
Suppose we have a score function t ( x ) ∈ R .
We classify x as positive if t ( x ) > τ , where τ is a threshold.
Let F + and F − denote the Cumulative Distribution Functions (CDFs) of t ( x + ) and t ( x − ) respectively.
Then we can define the following quantities:
• False Positive Rate (FPR) at threshold τ , ˜ F − ( τ ) :
= 1 − F − ( τ ) • True Positive Rate (TPR) at threshold τ , ˜ F + ( τ ) :
= 1 − F + ( τ ) •
The ROC curve of a score function t : the graph of function ˜ F + [ ˜ F − 1 − ( s )] , where s ∈ [0 , 1] .
The above definition of ROC curve requires F − to be strictly increasing.
In this paper, we assume
F + , F − to be both strictly increasing.
Obviously, both F + and F − depend on the choice of score function t .
2.3 Arc Length of ROC Curve
Due to the strict monotonicity of F + and F − , ˜ F + and ˜ F − form a bijective parameterization of the
ROC curve in the sense that each point on this ROC curve can be written as ( ˜ F − ( τ 0 ) , ˜ F + ( τ 0 )) for a unique τ 0 ∈ R .
Using the line integral formula, the arc length of an ROC curve for a fixed score function t can be expressed using the derivatives of ˜ F − and ˜ F + :
[Equation]
where f + ( τ ) and f − ( τ ) are the density functions of t ( x + ) and t ( x − ) respectively.
Although (1) is an elementary result, it has been seldom discussed in the ROC literature.
Authors in [3, 4] have proposed a performance metric computed over the “ROC hypersurface” and (1) is used to justify such a metric in a binary classification setting.
Using (1), we can confirm a simple geometric fact whose proof can be found in Appendix A:
Proposition 1. >
ROC( t ) ∈ [ √ 2 , 2] , for all t .
This result reflects the geometric observation that any monotone curve (such as ROC curve) starts and ends at two opposite corners of the ROC space [0 , 1] 2 has an arc length between √ 2 and 2 .
f -divergences Arising from ROC Arc Length 3.1 f -divergence of Score Distributions
Among many discrepancies measures, f -divergence has been widely used in many applications.
Definition 1.
Let p and q be densities of two continuous distributions.
An f -divergence is defined as:
D g ( p | q ) :=
E q (cid:104) g (cid:16) p ( z ) q ( z ) (cid:17)(cid:105) , where g is convex and lower-semicontinuous satisfying g (1) = 0 .
Now let us slightly rewrite (1).
Assuming f − is strictly positive (in which case F − is strictly increasing), we can write
[Equation]
[Equation]
where g ( s ) = √ s 2 + 1 −√ 2 .
Equation (2) yields the first important result of this paper: > ROC( t ) −√is an f -divergence between score densities f
+ and f − since g ( s ) is a convex function and g (1) = 0 .
This result confirms, for any given t , > ROC( t ) is a good discrepancy for measuring positive and negative scores (i.e., score distributions).
It also explains why the ROC arc length in Figure 1 is a good discrepancy measure: Given t ( x ) = x , the score distributions are same as the data distributions in each plot.
The arc length of ROCs in Figure 1 plots are f -divergences of the score distributions hence are also f -divergences of data distributions.
Although > ROC( t ) − √ 2 is an f -divergence of score distributions, it is not an f -divergence of the positive and negative data distributions for general t .
The choice t ( x ) = x in the toy example does not have simple analogues for higher dimensional datasets.
Are there choices of t such that the arc lengths of their ROC curves are good discrepancy measures on data distribution?
In what follows, we show when t is a bijective transform of the likelihood ratio p + ( x ) p − ( x ) , > ROC( t ) encodes the differences between p + ( x ) and p − ( x ) in the form of an f -divergence between p + ( x ) and p − ( x ) .
3.2 f -divergences of Data Distributions
Using the law of the unconscious statistician, we can express > ROC( t ) in terms of an expectation with respect to the negative data density p − ( x ) :
E p − (cid:114)(cid:104) f
+ ( t ( x ))
f − ( t ( x ))
(cid:105) 2 + 1 .
Consider a special family of score functions: t ∗ ( x ) = γ (cid:16) p + ( x ) p − ( x ) (cid:17) where γ is any strictly increasing function.
Due to the NeymanPearson lemma [25], ROC( t ∗ ) has the highest TPR at any FPR level.
Geometrically speaking, they dominate all other ROC cuves in an ROC plot and have the maximal AUC.
Hence, we refer to t ∗ as the optimal score and ROC( t ∗ ) as the optimal ROC curve.
For convenience, we denote the ROC( t ∗ ) as ROC ∗ which reads “rock star”.
It can be shown that
[Equation]
where the second equality holds due to (cid:82) D a ( x )d x (cid:82)
D b ( x )d x = γ − 1 ( C ) , when γ (cid:16) a ( x ) b ( x ) (cid:17) ≡ C, ∀ x ∈ D .
When γ ( x ) =
x , (3) expresses a known result [5], and is often given in plain English as “the density ratio of the likelihood ratio score is the likelihood ratio itself”.
Finally, > ROC ∗ takes an elegant form free from t ∗ or γ : E p − ( x ) (cid:114)(cid:104)
[Equation]
Equivalently
[Equation]
We can see that the same f -divergence arises from computing the arc length of ROC ∗ .
However, unlike the f -divergence given in (2), (4) is an f -divergence between data distributions, not score distributions.
It shows that as long as we use the the optimal scores, the arc length of the optimal
ROC can indeed reflect the differences between data distributions.
From now on, we will refer to >
ROC ∗ − √ 2 as the ROC divergence.
To the best of our knowledge, (4) has not been presented in literature before.
By definition, the ROC divergence is symmetric.
Moreover, as a result of Proposition 1, the ROC divergence is upper bounded by 2 − √ 2 and lower bounded by 0.
Some geometric properties of
ROC ∗ can be found in Section B.
Estimating the Arc Length of ROC ∗ 4.1 A Variational Objective
To numerically approximate the arc length using samples alone, we leverage that > ROC ∗ − √ 2 is an f -divergence.
Utilizing Fenchel’s duality [18], authors in [27] show that an f -divergence D g ( p + | p − ) has a variational representation:
[Equation]
where g ′ is the convex conjugate of g and the supremum is taken over all measurable functions.
In the case of the ROC divergence, g ( z ) = √ z 2 + 1 − √ 2 and z ∈ [0 , ∞ ] thus g has a convex conjugate g ′
( z ′ )
= − √ 1 − z ′ 2 − √ 2 , z ′ ∈ [0 , 1] .
Rewriting >
ROC ∗ − √ 2 using the above variational representation and dropping the −√ 2 , we obtain:
[Equation]
We reparameterize u ( x ) = sin[ v ( x )] , where v ∈ [0 , π/ 2] :
[Equation]
Differentiating the objective in (5) for v and setting the derivative to zero, we can see the supremum is attained at v ∗ =
atan p + p − .
In other words, the optimal v ∗ is the arctangent likelihood ratio function.
[Equation]
[Equation]
It is also interesting to see how v ∗ is visualized in the ROC plot.
We can see the tangent of ROC ∗ at an FPR level s 0 ∈ [0 , 1] is
[Equation]
where x 0 is any point in X that satisfies the equality γ (cid:16) p + ( x 0 ) p − (
[Equation]
0 ) (cid:17) =
˜ F − 1 − ( s 0 ) .
(6) is a known result [7].
In other words, v ∗ =
atan p + p − is the slope angle of ROC ∗ .
See Figure 2 for a visualization of the tangent of ROC ∗ expressed by the likelihood ratio.
Moreover, using (5), we can obtain a relationship between >
ROC ∗ and the total variation distance between P + and P − (denoted as TV( P + , P − ) ).
Proposition 2.
TV( P + , P − ) has a lower and upper bound expressed via > ROC ∗ :
[Equation]
Proof of this proposition can be found in Section C.
This proposition justifies that > ROC ∗ is a valid measure of the discrepancy between P + and P − .
We compare this bound with other known TV bounds in Section 6.1.
4.2 A Tractable Objective for Estimating atan p + p −
To use (5) in practice, we need to find an appropriate function class F .
We can simply restrict v to a bounded (parametric/non-parametric) function class F and solve the sample version of (5):
[Equation]
In practice, enforcing the boundedness v ∈ [0 , π/ 2] over X is difficult.
We can relax (7) by only enforcing the boundedness constraint of v on the sample dataset X + ∪ X − .
For example, by letting F be a Reproducing Kernel Hilbert Space (RKHS) [31], we can translate (7) into the following optimization problem:
[Equation] [Equation]
where H is a RKHS with a positive definite kernel k ( x , x ′ )
= ⟨ φ ( x ) , φ ( x ′ ) ⟩ , ∥ · ∥ H is the RKHS norm and λ 2 ∥ v ∥ 2 H is the regularization term.
The optimizer ⟨ ˆ v, φ ( x ) ⟩ is an estimation of v ∗ ( x ) , the arctangent of the likelihood ratio.
(8) is a strictly convex optimization and thus, if a solution ˆ v exists, it must be unique.
Instead of modelling atan p + p − , we can opt for modelling the log likelihood ratio log p + p − .
However, this modelling choice results in an non-convex optimization thus presents extra challenges in the theoretical analysis.
Details can be found in Section K.
4.3 Finite Sample Guarantee
We show that the solution of (8), ⟨ ˆ v, φ ( x ) ⟩ converges to the true arctangent likelihood ratio (or its projection onto H ) as the number of samples goes to infinity.
Below are a few regularity conditions.
Assumption 1.
There exists a unique v ∗ ∈ H , such that E [ ∇ v ℓ ( v ∗ )]
= 0 and ⟨ v ∗ , φ ( x ) ⟩ ∈ [0 , π/ 2] holds for all x ∈ X .
A sufficient condition of the above condition is specified in the following proposition.
Proposition 3.
If there exists a unique v ∗ ∈ H , such that ⟨ v ∗ , φ ( x ) ⟩ = atan (cid:104) p + ( x ) p − ( x ) (cid:105) then Assumption 1 holds.
The proof can be found in Appendix F. Proposition 3 states if model is correctly specified and identifiable then Assumption 1 holds.
It is possible that there exists a v ∗ ∈ H satisfying E [ ∇ v ℓ ( v ∗ )]
= 0 which does not meet the boundedness constraint [0 , π/ 2] .
In this paper we only consider situations where Assumption 1 holds, which includes all situations where the model is correctly specified and some situations where the model is misspecified.
Assumption 2.
Let n min =
min( n + , n − ) .
There exists a subspace H ∗ := { v ∈ H|∥ v − v ∗ ∥ 2 H ≤ δ 2 n min } , such that ∀ v ∈ H ∗ , ∀ x ∈ X + ∪ X − , ⟨ v, φ ( x ) ⟩ ∈ (0 , π 2 ) holds with high probability.
The sequence δ n min is monotonically decreasing as n min grows to infinity.
Assumption 2 states all v within a vicinity of v ∗ are in the interior of (8)’s feasible region with high probability.
The following proposition gives a sufficient condition under which Assumption 2 holds.
Proposition 4.
Suppose ∥ φ ( x ) ∥ H ≤ 1 .
If our model is correctly specified as described in Proposition 3 and ∀ x ∈ X , atan (cid:104) p + ( x ) p − ( x ) (cid:105) ∈
[ R 1 , R 2 ] , for some R 1 and R 2 such that π 2 > R 2 > R 1 > 0 , then there exists N > 0 such that Assumption 2 holds when n min > N .
The proof can be found in Appendix D. Since we use RKHS as the estimator function class, our final assumption is that v ∗ should be reasonably smooth.
In previous works such an assumption depends on the decay of the integral operator’s eigenvalues [34, 8].
In this paper, we measure the smoothness using the range space technique which has been recently adopted in [9, 33].
We define
[Equation]
where ⊗ denotes the outer product.
Given v 0 ∈ H , Σ v 0 is an integral operator on u ∈ H and
[Equation]
By definition Σ v 0 is a positive, self-adjoint operator, in the sense that ⟨ u, Σ v 0 u ⟩ ≥ 0 , ⟨ u, Σ v
[Equation]
⟩ = ⟨ Σ v 0 , u, v ⟩ , ∀ v, u ∈ H .
Moreover, some algebra shows that Σ v 0 is also a bounded and compact operator.
See Section G for more details.
Next, we assume the true arctangent ratio function (or its projection) is in the range space of Σ v ∗ .
Assumption 3.
Let R ( Σ v ∗ ) denote the range space of Σ v ∗ .
There exists 0 < β ≤ 1 , v ∗ ∈ R ( Σ βv ∗ ) , where C β is the fraction power of a compact, positive and self-adjoint operator C .
Note that the larger β is, the smoother the functions in the range space are.
More discussions on the range space assumption can be found in Section 4.2, [33].
Now we are ready to state our theorem:
Theorem 1 (Convergence Rate of ˆ v ) .
Suppose Assumptions 1, 2 and 3 hold and ˆ v exists.
If ∥ φ ( x ) ∥
H ≤ 1 and
[Equation]
where B + = ∥ ( Σ v ∗ + λ I ) − 1 E p + [ φ ( x )]
∥ H , B − =
∥ ( Σ v ∗ + λ I ) − 1 E p − [ φ ( x )]
∥ H and T ≥ 1 , K > 0 are constants that do not depend on n min , then there exists a constant N > 0 such that when n min > N , ∥ ˆ v − v ∗ ∥
H
=
O p ( n − β/ 4 min ) .
The proof can be found at Appendix E. Theorem 1 shows that, under mild assumptions, ˆ v is indeed a good estimator for atan (cid:104) p + ( x ) p − ( x ) (cid:105) .
Some discussions comparing our results with convergence results proved by Nguyen et al.
[26] can be found in Section M. Since atan (cid:104) p
+ ( x ) p − ( x ) (cid:105) is an optimal score that gives rise to ROC ∗ , our estimator may have some interesting applications such as outlier detection [17] or Neyman-Pearson classification [35].
We will defer discussions on those applications in future works.
In the next section, we employ our arctangent likelihood ratio estimator in an application of lower bounding the maximal AUC.
Approximately Lower Bounding the Maximal AUC
Finding a score function t that approximately maximizes AUC is an important task in binary classification.
Let us denote the AUC of ROC( t ) as AUC( t ) .
It can be seen that
[Equation]
Due to the Neyman-Pearson lemma, ROC ∗ has the maximum AUC among all ROC curves.
Denote the AUC of ROC ∗ as AUC ∗ Consider the following inequalities:
[Equation]
where L ( a, b ) is a continuous and concave lower bound of the indicator function 1 ( a > b ) .
Due to the Neyman-Pearson lemma, the supremum of (i) is only attained when t ( x ) = γ (cid:16) p + ( x ) p − ( x ) (cid:17) where γ is a strictly increasing function.
Replacing the expectations in (ii) with sample averages yields the optimization problem of AUC maximization [2, 10]:
[Equation]
The objective above is also referred to as Wilcoxon-Mann-Whitney statistic [14].
Therefore, we can see that AUC maximization is a procedure that approximates an optimal score function by maximizing a lower bound of AUC ∗ .
Now, we show a different way of lower bounding AUC ∗ with the help of atan p + ( x ) p − ( x ) .
We have seen that how ( ˜ F − ( τ ) , ˜ F + ( τ )) parameterizes an ROC curve in Section 2.3.
In fact, the area between
ROC ∗ and the diagonal line from (0 , 0) to (1 , 1) can be similarly parameterized by considering ROC curves of positive and negative mixture score distributions.
Let F ∗ + and F ∗− denote CDFs of any optimal score.
For α ∈ [0 , .
5] , we can define CDFs of α -mixtures of F ∗ + and
F ∗− as follows
[Equation]
Then, FPR ( ˜ F ∗− ( τ, α ) ) and TPR ( ˜ F ∗ + ( τ, α ) ) for these αmixtures can be defined accordingly.
We visualize densities of F ∗ + ( α ) and F ∗− ( α ) for different α in Figure 3.
Further, we can see that the 2-dimensional coordinate r ( τ, α ) :
[Equation]
[Equation] [Equation]
, α ) , ˜ F
[Equation]
+ (
[Equation]
, α )) parameterizes the area between ROC ∗ and the diagonal line in [0 , 1] 2 :
• When fixing α and varying τ , the coordinates give rise to a smooth curve in ROC space from [0 , 0] to [1 , 1] .
– When α = 0 , such a curve is ROC ∗ .
– When α = .
5 , such a curve is the diagonal line.
• When fixing τ = τ 0 and varying α , the coordinates produce a straight line segment connecting ( ˜ F ∗− ( τ 0 , 0) , ˜ F ∗ + ( τ 0 , 0)) and ( ˜ F ∗− ( τ 0 , . 5) , ˜ F ∗ + ( τ 0 , . 5)) .
1.2 FPR0.
T P R 0.
1 FPR 0.
T P R 0.
1 FPR 0.
T P R
The left plot in Figure 4 visualizes this parameterization.
Now the surface area sandwiched between
ROC ∗ and the diagonal line can be computed using a surface integration:
[Equation] [Equation]
where × denotes the cross product.
After some algebra and applying the Fenchel duality technique in
Section 4.1, we prove that AUC ∗ can be expressed as the supremum of a variational objective similar to (5):
Proposition 5.
AUC
[Equation]
where w ( τ ) := sin( τ + π 4 ) · | F ∗ + ( τ ) − F ∗− ( τ ) | .
The supremum of (12) is attained at v ∗ =
atan p + p − .
The proof can be found in Appendix H in the supplementary material.
A lower bound of A can be obtained by restricting v to a function class.
Evaluating w requires us to evaluate atan p + ( x ) p − ( x ) , F ∗ + and F ∗− which are not readily available.
However,
Section 4.3 shows that the empirical estimator (8) is a consistent estimator of atan p + ( x ) p − ( x ) under mild conditions.
Therefore, we propose the following two-step procedure to approximately lowerbound A :
Algorithm
Two-step Procedure for
Approximately
Lower Bounding A
Obtain ˆ t ( x ) := ⟨ ˆ v, ϕ ( x ) ⟩ using (8).
Approximate F ∗ + and F ∗− using ˆ F + and ˆ F − which are empirical CDFs of ˆ t ( x + ) and ˆ t ( x − ) .
Optimize the empirical version of (12) by restricting v to a feasible function class and plugging in estimates obtained in the earlier step, i
e.,
[Equation]
13) where ˆ w ( τ ) :
= sin( τ + π 4 ) (cid:12) (cid:12) (cid:12) ˆ
[Equation]
−
[Equation]
[Table]
Note that (13) is nothing but a weighted sample objective (7).
Thus, it can be easily optimized by the algorithm that solves a weighted version of (7) given the approximated weights in the first step.
In practice, we simply run the solver for (8) twice:
The first time we run it without weights then run it again with weights ˆ w (cid:2) ˆ t ( x i ) (cid:3) calculated from the first run.
Since the above algorithm also approximates an optimal score ( atan p + p − ) by maximizing an approximated lower bound of AUC ∗ , it is natural to wonder how the maximizer ˆˆ v of (13) would perform in AUC maximization tasks.
In the next section, we show that our two-step algorithm achieves a promising AUC performance compared to a state of the art AUC maximizer.
Computing AUC ∗ using (11) is different from using Wilcoxon-Mann-Whitney statistic (i.e., (13)):
Our approach divides the space between ROC ∗ and the diagonal into small surface elements and then adds them up.
Wilcoxon-Mann-Whitney statistic adds up all histogram bars, which are TPRs at different FPR levels.
Our approach requires ˜ F + and ˜ F − to be differentiable with respect to τ , which means the score distributions cannot be discrete.
However, Wilcoxon-Mann-Whitney can compute the AUC of discrete score distributions without a problem.
This difference is visualized in the middle and right plots of Figure 4.
Experiments 6.1 Numerical Comparison of Divergences and TV Bounds
In this experiment, we numerically compare the ROC divergence, the upper and lower bound in
Proposition 2 with several other divergences and some known bounds of TV in Figure 5.
In this numerical simulation, p + = N (0 , 1) and p − = N ( δ, 1) .
We plot ROC divergence, Jensen Shannon divergence, Wasserstein distance and TV between p + and p − as δ grows from 0 to 5.
We can see the (rescaled) ROC divergence closely resembles TV .
When δ > 1 . 45 , the upper bound given in
Proposition 2 is the tightest among known TV upper bounds (Pinsker’s upperbound, Bretagnolle &
Huber’s upper bound, Le Cam’s upper bound).
This suggests that combining our upperbound with existing bounds may produce an even tighter bound for TV .
6.2 Imbalanced Classification on CIFAR-
In this section, we test if the ˆˆ v obtained in our two-step procedure (13) is indeed a good score function in terms of AUC in imbalanced classification tasks.
We use a widely known image classification dataset CIFAR-10 [20].
The performance is compared with an AUC maximizer which maximizes the empirical lower bound in (10) and a vanilla logistic regression classifier.
We set the surrogate loss L ( a, b ) :
= − (1 − ( a − b )) 2 in the AUC maximizer, as suggested in [10].
All methods use linear models with no regularization terms since our models are simple and we have sufficient samples.
Particularly, the ˆ t and ˆ ˆ v in our two-step algorithm is obtained using (8) by setting φ ( x ) = x and λ = 0 .
The AUC maximization (AUC max) is implemented using SPAUC method [22].
Instead of using the raw features, we extract 50 dimensional bounded features by training a residual network [15] on the training dataset using the 10-class cross entropy loss.
The structure of the network is included in the supplementary material.
After obtaining features, we construct datasets for different one-versus-the-rest classification tasks.
For a single task, we pick a class and obtain X + by randomly sampling from this class n + times in the training set.
Similarly, X − is obtained by randomly sampling from the rest of the classes n − times.
In our experiments, we set n + = 24 , 48 , 72 , 96 ,and fix n − = 1000 to create imbalanced positive and negative datasets.
We run all three methods and obtain the corresponding score functions.
For each class, we repeat the experiment 96 times using different random samples.
We use the testing and training split provided by the dataset itself.
Our experiments can be seen as a transfer learning task which reuses predictive features trained for a multi-class classifier for one-versus-the-rest binary classification tasks.
The average AUCs computed on the testing dataset and their standard errors over 96 runs over different n + sample sizes are shown in Figure 6.
Our method has approximately equal performance with the AUC maximizer despite not directly maximizing the AUC.
This observation indicates that ˆˆ v can be a good score function in AUC maximization tasks.
Both of the methods significantly outperform vanilla logistic regression.
6.3 Discussions on Computational Complexity
Without loss of generality, assume n + = n − = n .
The naive caclulation of objective (10) has a computational complexity O ( n 2 ) since we evaluate the loss function L at each pair of samples.
However, authors in [19] have shown that the objective function in (10) can also be computed with
O ( n log( n )) complexity for hinge loss (and decomposable loss functions).
A recent work [37] simplifies the computation of (10) for the squared loss function L with an unbiased estimate. Suppose t ( x ) := ⟨ v, x ⟩ , then the negative objective of (10) is an unbiased estimate of [Equation] After we approximating (14) with empirical terms, the computation can be done with a complexity
O ( n ) .
When implemented in an online fashion, it has a computational complexity of one datum.
In comparison, the objective (7) and (13) are summation of sin / cos( v ( x )) evaluated at each datum, so computing the objective/gradient has a computational complexity O ( n ) .
Computing ˆ F + and ˆ F − requires sorting our dataset, which has an average complexity O ( n log n ) .
However, once our datasets are sorted, ˆ F + (cid:0)
ˆ t ( x 0 ) (cid:1) =
in + , where i is the index of ˆ t ( x 0 ) in the sorted set { ˆ t ( x i ) } n + i =1 .
Acknowledgments and Disclosure of Funding
The author would like to thank Prof. Peter Flach and Dr. Hao Song for their helpful discussions.
The author would like to thank four anonymous reviewers for their insightful comments.
In particular, we thank Reviewer WSBr and the Area Chair for pointing out the computational complexity inaccuracies in our initial version.
A Proof of Proposition
Proof.
Jensen’s inequality:
[Equation]
B Geometric Properties of ROC ∗
Here we prove a result regarding some other geometric properties of ROC ∗ .
Proposition 6.
ROC ∗ is a convex curve and > ROC ∗ is the longest among all convex ROC curves.
Proof.
First, we show ROC ∗ is a convex curve.
To show ROC ∗ is convex, we only need to show ˜
F + ( ˜ F − 1 − ( s )) is a concave function.
This can be verified by checking the sign of ∂ 2 s ˜ F + ( ˜ F − 1 − ( s )) :
[Equation]
where the second equality is due to (3) and x 0 is any point in X that satisfies the equality γ (cid:16) p +
[Equation]
x 0 ) p − ( x 0 ) (cid:17) =
˜ F − 1 − ( s ) .
Further, we can show that,
[Equation]
Since γ is a strictly monotone increasing function, the first factor is non-negative and the second factor is also strictly positive due to the our assumption on the positivity of f − .
We have ∂ 2 s F + ( F − 1 − ( s ; t ∗ )) ≤ 0 .
Moreover, at any FPR level s ∈ [0 , 1] , the Neyman-Pearson lemma [25] implies
[Equation]
where ˜ F ′ + and ˜ F ′− are TPR and FPR of any other score function.
In words, ROC ∗ dominates all other ROC curves.
Since ROC ∗ is convex and encloses all other ROC curves, our claim follows
Archimedes’s Second Axiom: among all convex curves with the same endpoints, the one encloses all other curves has the longest arc length.
C Proof of Proposition
Proof.
Using the integral probability metric representation of TV( P + , P − ) [32], we can write:
[Equation]
Some algebra can show that z ≥ sin( z )
a
[Equation]
and − z ≥ cos( z ) a − arcsin( a ) − √ 1 − a 2 a for all a ∈ [0 , 1] and z ∈ [0 , π/ 2] .
Therefore
[Equation]
Similarly, multiplying both sides of the second equality above by 2 π , we obtain
[Equation]
D Proof of Proposition
Proof.
∀ v
[Equation]
H
[Equation]
, |
[Equation] [Equation]
, φ
[Equation] [Equation]
| ≤ ∥
[Equation]
− v ∗ ∥
H ∥ φ ( x ) ∥
H ≤ δ n min .
If δ n min < min( R 1 , π 2 −
R 2 ) then ⟨ v, φ ( x ) ⟩ ∈ (0 , π 2 ) holds uniformly for every x ∈ X .
As δ n min is a decaying sequence, there always exists an N such that δ n min ≤ min( R 1 , π 2 − R 2 ) holds for n min ≥ N .
E Proof of Theorem
To reduce the visual clutter, in this section, ∥ v ∥ represents the Hilbert space norm of v , defined as (cid:112) ⟨ v, v ⟩ .
We simplify E p + [ v ( x )] as E + [ v ( x )] whenever it does not lead to confusion.
For ease, we write (cid:80) n + i =1 f ( x + i ) as (cid:80) n + i =1 f ( x i ) , a convention which will adopted henceforth.
Proof.
Define H ∗ := { v ∈ H|∥
v − v ∗ ∥ 2 ≤ δ 2 } .
Consider an optimization that is similar to (8):
[Equation]
Define ˜ u :
=
˜ v − v ∗ and we have the following equality due to the KKT conditions of (15)
[Equation]
where ν is a Lagrangian multiplier and ν ≥ 0 .
Multiplying both sides by ˜ s
= ( Σ v ∗ + λ I ) − 1 ˜ u , we have
[Equation]
Let g ( v ) :
= ⟨ ˜ s, ∇ v ℓ ( v ) + λv + 2 ν ( v − v ∗ ) ⟩ , we can applying Mean Value Theorem (MVT) on the scalar valued function g ( v ) :
[Equation]
16) where ¯ v =
av ∗ + (1 − a )
ˆ v for some a ∈ [0 , 1] .
Knowing g
(˜ v ) =
and g ( v ∗ ) = ⟨ ˜ s, ∇ v ℓ ( v ∗ )
+ λv ∗ ⟩ , we can translate (16) into
[Equation]
where I is the identify matrix.
Focusing on the RHS, we have ⟨
[Equation]
[Equation]
The first line is due to the fact that ⟨ 2 ν ˜ s, ˜ u ⟩ ≥ 0 .
Use the inequality (18) on (17), we get the inequality ⟨ ˜ s, −∇
[Equation]
First, let us inspect a .
Using MVT on sin ⟨ v, φ ( x ) ⟩ , v ∈ H ∗ and applying Hölder’s inequality, we get
[Equation]
Define Σ + v := E + [sin ⟨ v, φ ( x ) ⟩ φ ( x ) ⊗ φ ( x )] and ˆ Σ
+ v as its empirical counterpart approximated using X + .
We can see that a = ⟨ ˜ s, [ Σ + v ∗ − Σ +¯ v ]˜ u ⟩ + ⟨ ˜ s, [ Σ − v ∗ − Σ − ¯ v ]˜ u ⟩ .
Moreover,
[Equation]
( i ) is due to (20).
Following a similar line of reasoning, we can see
[Equation]
By setting δ ≤ 4 max (cid:16) ∥ (
[Equation] [Equation]
+ [
[Equation]
[Equation]
)] ∥ , ∥ ( Σ v ∗ + λ I ) − 1 E − [ φ ( x )] ∥ (cid:17) − 1 , we have
[Equation]
Now we inspect b .
We can see | b | ≤ (cid:12)(cid:12)(cid:12)
[Equation]
ˆ
[Equation]
u
[Equation]
u
[Equation]
+
[Equation]
Define a scalar random variable
Z
[Equation]
i
[Equation] [Equation]
By definition 1 n + (cid:80) n + i =1 Z ( i ) f
= ˜ s ⊤ ˆ Σ + f ˜ u.
Therefore
[Equation]
Since 0 ≤ Z ( i ) v ≤ ∥ ˜ s ∥ · ∥ ˜ u ∥ · ∥ φ ( x ) ∥ 2 ≤ ∥ ( Σ v ∗ + λ I ) − 1 ˜ u ∥∥ ˜ u
∥ ≤ ∥ ˜ u
∥ 2 λ , using Uniform Law of
Large Number for bounded random variable (Theorem 4.10, [36]),
[Equation]
i
[Equation]
with high probability, where R n + ( F Z ) is the Rademacher complexity of the function class of Z v .
It remains to bound R n + ( F Z ) .
It can be seen that Z f = h [ ⟨ v, φ ( x ) ⟩ ] where h is a Lipschitz continuos function with Lipschitz constant ∥ ˜ u
∥ 2 λ .
Hence, due to Ledoux–Talagrand contraction inequality (see, e.g., (5.61) in [36]), R n + ( F Z ) is upperbounded by,
[Equation]
where C 0 is a universal constant.
The last inequality is due to Corollary 14.5 in [36].
Therefore
[Equation]
and similarly,
[Equation]
Therefore,
[Equation]
with high probability.
Substituting (21)and (22) into (19), we get
[Equation]
Using triangle inequality and Hölder’s inequality
[Equation]
we have
[Equation]
[Equation]
Due to Assumption 1, E [ ∇ v ℓ ( v ∗ )]
= 0 .
Hence,
[Equation]
We can see
[Equation]
holds with high probability and C 1 is a universal constant (due to Lemma 2)
[Equation]
Moreover, since v ∗ ∈ R ( Σ βv ∗ )
[Equation]
there exists g ∈ H
[Equation]
v ∗ =
Σ βv ∗ g .
Notice Σ v ∗ is a bounded, compact, self-adjoint linear operator (see Section G).
Therefore, Hilbert-Schmidt Theorem indicates, Σ v
[Equation]
(cid:80) i
α
i ψ i ⟨ ψ i , ·⟩ , where ψ i , α
i are eigenfunctions and eigenvalues of Σ v ∗ respectively.
Hence,
[Equation]
i
[Equation] [Equation]
Combine (23), (24) and (25) and cancel ∥ ˜ u ∥ , we can conclude that
[Equation]
with high probability.
Set λ = max( C 1 ,C 0 , 1) n 1 /min , we have
[Equation]
holds with high probability.
Therefore, ∃ N 2 , when n min > N 2 , ∥ ˜ u
∥ =
O p ( n − β/ 4 min ) .
Since ∥ ˜ u ∥ = o p (1) , as long as δ ≥ K · n − β/ 4 min where K > 0 is a constant, there exists a constant N such that, when n min > N , ˜ v is in the interior of H ∗ with high probability.
When this happens, the constraint v ∈ H ∗ is no longer active.
This means ˜ v is a stationary point of the objective function in (15).
Moreover, ˜ v ∈ H ∗ , so it is in the feasible region of (8) thanks to Assumption 2.
This further indicates that ˜ v is also a solution to (8).
As (8) is a strictly convex optimization problem, ˜ v is also its only solution.
Therefore ˜ v =
ˆ v and ∥
ˆ v − v ∗ ∥ =
∥
˜ v − v ∗ ∥
=
O p ( n − β/ 4 min ) .
Lemma 2.
Given any v ∗ ∈ H such that E [ ∇ v ℓ ( v ∗ )]
= 0 , if ∥ φ ( x ) ∥
H ≤ B then
[Equation]
exp (cid:18) − n min δ
B(cid:19) .
Proof.
Write down the definition of ∇ v ℓ ( v ∗ ) .
Notice ∇ v ℓ ( v ∗ ) = − 1 n + n + (cid:88)
i = cos ⟨ v, φ ( x i ) ⟩ φ ( x i ) (cid:124) (cid:123)(cid:122) (cid:125) a +n − n − (cid:88)
i = sin ⟨ v, φ ( x i ) ⟩ φ ( x i ) (cid:124) (cid:123)(cid:122) (cid:125) b .
By using Hilbert-space Hoeffding’s inequality [30], we know for all δ a , δ b >
P ( ∥ a − E
[ a ] ∥ H > δ a ) ≤ 2 exp (cid:18) − Cn + δ 2 a
B(cid:19) and P ( ∥ b − E [ b ] ∥ H > δ b ) ≤ 2 exp (cid:18) − Cn − δ 2 b
B(cid:19) , where C is a constant.
Let δ =
δ a = δ b ,
[Equation]
where the first equality used the condition that E [ ∇ v ℓ ( v ∗ )]
=
E [ a ] + E
b ] = 0 .
This completes the proof.
F Proof of Proposition
Proof.
We start from the definition of E [ ∇ v ℓ ( v ∗ )] :
[Equation]
where the third equality is due to the fact that ⟨ v ∗ , φ ( x ) ⟩ = atan p + ( x ) p − ( x ) .
Since p + /p − ∈ [0 , ∞ ) , ⟨ v, φ ( x ) ⟩ ∈ [0 , π/ 2) .
As v ∗ is unique by assumption, Assumption 1 holds.
G Properties of Operator Σ v
By construction, it is easy to verify that Σ v 0 is self-adjoint.
First, we prove that the integral operator
[Equation]
, is a bounded operator.
For all u ∈ Ball(0 , 1) , where Ball(0 , 1) is the unit ball in ∥ · ∥ H , ∥
[Equation]
Hence, Σ v 0 is a bounded operator as long as ∥ φ ( x ) ∥ H is bounded.
Second, we show Σ v 0 is trace class hence compact.
Let ψ i , i ∈ N be an orthonormal basis in H , then (cid:88)
[Equation]
∞ holds as long as ∥ φ ( x ) ∥ H is bounded.
This shows Σ v 0 is trace-class and therefore, compact.
H Proof of Proposition
Proof.
Let us define for α ∈ [0 , .
[Equation]
We can see that r ( τ, α ) :
= ( ˜ F ∗− ( τ, α ) , ˜ F ∗ + ( τ, α )) is a parameterization for the space between ROC ∗ and the diagonal from (0 , 0) to (1 , 1) .
We can compute the surface area using the surface integral formula:
[Equation]
where ∂ τ r ( τ, α )
=
[Equation]
∗− ( τ, α ) ∂ τ ˜ F ∗ + ( τ, α ) 
 and ∂ α r ( τ, α ) =
 
∂ α ˜ F ∗− ( τ, α ) ∂ α ˜ F ∗ + ( τ, α )   .
It can be seen that ∂ α r ( τ, α ) = (cid:34) F ∗− ( τ ) − F ∗ + ( τ )
F ∗ + ( τ ) − F ∗− ( τ ) (cid:35) for all α .
Rewrite A 0 :
A 0 = (cid:90) dom( τ ) (cid:90) [
[Equation]
[Equation] [Equation]
where a ( τ ) =
[Equation]
and b
[Equation]
=
[Equation]
Both a and b are free from α .
Rewriting the cross product in (26) in a different form
[Equation]
we obtain
[Equation] [Equation]
where θ ( τ ) is the angle between a ( τ ) and b ( τ ) .
b ( τ ) is the tangent vector of the ROC ∗ .
Knowing the slope of ROC ∗ is the likelihood ratio (see Section 4.1) and a ( τ ) points at the 45 degree downward regardless of τ , we can see θ ( τ ) =
(cid:104) atan p ( x ) q ( x ) (cid:105)
+ π 4 .
Using the fact that f ∗ + ( t ( x ))
f ∗− ( t ( x ))
=
p + ( x ) p − ( x ) and the law of unconscious statistician:
[Equation]

[Equation]
Replacing (cid:114) p + ( x ) p − ( x ) 2 + 1 with its Fenchel dual as introduced in Section 4.1 and pulling the sup out of the expectation yields the desired result.
Differentiating the objective (12) with respect to v and setting the derivative to zero, we can see that superemum is attained at v ∗ =
atan p + p − .
I Wall Clock Comparison
In this experiment, we evaluate the computation time of the our two-step algorithm and the naive implementation of the offline AUC maximization (10) by plotting the wall clock time in Figure 7.
Both the AUC maximization and two-step procedure are implemented using MATLAB’s optimization toolbox.
See Section J for details.
The two-step procedure’s computation time grows at a much slower rate than the offline AUC maximization via a pairwise loss function.
Note that as we explained in
Section 6.3, if the surrogate loss is decomposable, the objective can be computed with a computational complexity O ( n log( n ))
If it the loss is squared loss, the offline algorithm can be performed with a O ( n ) computational complexity [37].
In this experiment, both methods are written in fully vectorized code.
The first order derivatives are provided to the fmincon and fminunc to accelerate the computation.
Code can be found in the supplementary material.
J Experiment Setup
In Section 6.2, we reduce the dimension of CIFAR-10 dataset to 50.
We first train a residual neural network [15] using logistic regression on all 10 classes.
This 103-layer network structure was borrowed from a MATLAB tutorial ( https://www.mathworks.com/help/deeplearning/ug/ train-residual-network-for-image-classification.html ).
MATLAB provides a pretrained version of this network.
To obtain bounded features, we append a fully connected linear layer (output dimension 50) and a bounded activation layer (clipped-relu) to the last average pooling layer in the network.
We freeze the earlier layers and only train the last two layers for 5 epochs.
The dataset and the code that reproduces Figure 6 can be found in the supplementary materials.
We invite reviewers to reproduce our results.
[Equation]
We can also leverage that v ∗ is the arctangent of the likelihood ratio and introduce mild assumptions on p + and p − .
When p + ( x ) and p − ( x ) are both members of the exponential family and share the same sufficient statistic h ( x ) ∈ R m , then ∃ v ∗ ∈ R m such that log (cid:104) p + ( x
[Equation]
p − ( x
[Equation]
(cid:105)
[Equation]
v
[Equation]
, h ( x )
[Equation]
+ C , where C is a constant.
If we choose to parameterize the log likelihood ratio using a linear model, ⟨ v , h ( x ) ⟩ + v 0 , then (7) becomes (ˆ
[Equation]
argmax
[Equation]
- -    x 0.
ProposedGround Truth
Note we do not have to restrict the optimization to a bounded function family as log (cid:104) p + ( x ) p − ( x ) (cid:105) ∈ R , and ⟨ ˆ v , h ( x ) ⟩
+ ˆ v 0 is an estimate of the likelihood ratio.
In this paper, we focus on (8) since the objective in (28) is non-convex with respect to v thus presents extra challenges in the theoretical analysis, although (28) is easier to implement in practice due to its unconstrained nature.
L Numerical Simulation of atan p + p − Estimation
We draw 100 samples from X + ∼ N (1 , 1) and X − ∼ N ( − 1 , 1) and solve (8) to estimate the arctangent density ratio.
The estimated arctangent density ratio with standard deviation (overruns) are plotted in Figure 8.
We use Gaussian kernel and hyperparameters (kernel bandwidth and regularization parameter) are tuned using cross validation.
We observe that the estimated arctangent ratio using the proposed method is very close to the ground truth and has a small standard deviation.
M Comparison with Convergence Result in Nguyen et al. [26]
The convergence of (log) density ratio estimation have been developed for two KL divergence based estimators [26].
However, these convergence theories are not general theories for arbitrary f -divergences.
Thus their proofs cannot be easily applied to our ROC divergence.
Moreover, Theorem 1 is not a minor modification of convergence theories in [26].
Specifically,
Nguyen et al.
[26] prove the likelihood ratio converges in Hellinger distance, while we prove the arctangent likelihood ratio converges in Hilbert space norm.
The proofs rely on completely different machinery and assumptions.
These technical results depend on the variational objective functions the estimators maximize and are not interchangeable.