Estimating the Arc Length of the Optimal ROC Curve and Lower Bounding the Maximal AUC
Anonymous Author(s)
AffiliationAddress email
Abstract
In this paper, we show the arc length of the optimal ROC curve is an f -divergence.
By leveraging this result, we express the arc length using a variational objective andestimate it accurately using positive and negative samples.
We show this estimatorhas a non-parametric convergence rate O p ( n − β/ 4 ) ( β ∈ (0 , 1] depends on thesmoothness).
Using the same technique, we show the surface area sandwichedbetween the optimal ROC curve and the diagonal can be expressed via a similarvariational objective.
These new insights lead to a novel two-step classificationprocedure that maximizes an approximate lower bound of the maximal AUC.
Experiments on CIFAR-10 datasets show the proposed two-step procedure achievesgood AUC performance in imbalanced binary classification tasks while being lesscomputationally demanding than the classic AUC maximizer in the offline setting .
Introduction
The study of Receiver operating characteristic (ROC) curves has a long history in medicine [23],psychology [13] and radiology [14, 11].
In machine learning, ROC curves have been primarily usedto analyze the performance of different classification algorithms [6, 7].
Indeed, the Area Under the
Curve (AUC) encodes the ranking accuracy of a classifier, which makes it a preferable performancemetric for imbalanced class classification [6, 2].
In recent years, ROC curves have also been used inapplications comparing two distributions and achieves promising results.
Examples include analyzingthe mode collapsing issue of Generative Adversarial nets (GAN) [22], and diagnosing the performanceof an amortized Markov Chain Monte Carlo [16].
In applications that require computing statistical discrepancy between distributions (e.g. GAN [12] or
Variational Inference (VI) [1]), f -divergences are a widely used discrepancy measures.
The family off -divergences includes Kullback-Leibler divergence [20] and Total Variational distance.
It has beenshown that f -divergences in general can be expressed via variational objectives and can be efficientlyapproximated from empirical samples [26, 27].
One might question that as the ROC curves are used as performance metrics in many two sampleapplications, are they in any way related to f -divergences?
For example, can AUC be an f -divergencebetween positive and negative data distributions given some classification score function?
Anearlier investigation proves that the answer is no when the score function is the likelihood ratio [28].
Nonetheless, this result inspired us to look for f -divergence from other geometries of the ROC curve.
In this paper we show that, when using the likelihood ratio score, a novel f -divergence arisesfrom computing the arc length of the corresponding ROC curve.
By leveraging this result, we canexpress the arc length using a variational objective and approximate it using only samples from twodistributions.
We show this arc length estimator is also a consistent estimator to the arctangent of
[Figure]
likelihood ratio and has a non-parametric convergence rate O p ( n − β/ 4 ) , where β ∈ (0 , 1] depends onthe smoothness of the true arctangent likelihood ratio.
Moreover, by parameterizing the ROC curvesof mixtures of positive and negative distributions, the surface area sandwiched between the optimal
ROC curve and the diagonal can be expressed via a similar variational objective.
With the help of ourarctangent ratio estimator, we can approximately maximize a lower bound to this surface area.
Wepoint out the similarity between this lower bound maximization and the classic AUC maximization[2] and show our approximated optimal score achieves comparable performance to an offline AUCmaximizer in an imbalanced classification problem while being less computationally demanding.
Background2.1 An Illustrative Example
ROC curves are frequently used to visualize binary classification performance, and we often rely onthe Area Under the Curve (AUC) as a numerical metric for selecting a good classifier.
In this section,we highlight an often overlooked ROC geometry: arc length.
We illustrate its potential as a gooddiscrepancy measure between positive and negative data distributions.
Let us consider the one-dimensional distributions and the ROC curves in Figure 1.
In this example,
ROC curves are generated using the identity score function t ( x ) = x .
In both (a) and (b), p + and p − are quite different, and thus the discrepancies between positive andnegative distributions should be large in both cases.
In (c), the densities p + and p − are the sameand consequently their discrepancy should be smaller than that of both (a) and (b).
Further, noticethat both (a) and (b) have long ROC curves ( ≈ 2 ) while (c) has a shorter one ( ≈ √ 2 ).
This exampletherefore suggests that the more similar p + and p − are, the shorter the ROC curve is.
However, we can see that t ( x ) = x is a special choice: If t ( x ) = 0 , the arc length will notreflect any discrepancy between data distributions at all.
This simple observation promotes thefollowing questions: Why is the arc length in this example good at telling the differences betweentwo data distributions?
Are there other score functions whose corresponding ROC arc length aregood discrepancy measures between positive and negative data distributions?
What are the practicalapplications of studying the arc length of ROC?
In the following sections, we study the arc length ofan ROC curve under a probabilistic framework and provide answers to these questions.
2.2 ROC Curve in a Probabilistic Setting
Suppose we have positive and negative datasets X
+ :=
{ x + i } n + i =1 and X − := { x − i } n − i
=1 drawn fromtwo distributions P + and P − respectively.
These distributions have respective probability densityfunctions p + ( x ) and p − ( x ) that are both defined on the domain X ⊆ R d .
A classification scorefunction (score function for short) takes a sample x as input and outputs a real-valued score.
Supposewe have a score function t ( x ) ∈ R .
We classify x as positive if t ( x ) > τ , where τ is a threshold.
Let F + and F − denote the Cumulative Distribution Functions (CDFs) of t ( x + ) and t ( x − ) respectively.
Then we can define the following quantities:
• False Positive Rate (FPR) at threshold τ , ˜ F − ( τ ) :
= 1 − F − ( τ )
• True Positive Rate (TPR) at threshold τ , ˜ F + ( τ ) :
= 1 − F + ( τ )•
The ROC curve of a score function t : the graph of function ˜ F + [ ˜ F − 1 − ( s )] , where s ∈ [0 , 1] .
The above definition of ROC curve requires F − to be strictly increasing.
In this paper, we assume
F + , F − to be both strictly increasing.
Both F + and F − depend on the choice of score function t .
2.3 Arc Length of ROC Curve
Due to the strict monotonicity of CDFs, ˜ F + and ˜ F − form a bijective parameterization of the ROCcurve in the sense that each point on this ROC curve can be written as ( ˜ F − ( τ 0 ) , ˜ F + ( τ 0 )) for a uniqueτ 0 ∈ R .
Using the line integral formula, the arc length of an ROC curve for a fixed score function tcan be expressed using the derivatives of ˜ F − and ˜ F + :
[Equation]
where f + ( τ ) and f − ( τ ) are the density functions of t ( x + ) and t ( x − ) respectively.
Although (1)is an elementary result, it has been seldom discussed in the ROC literature.
Authors in [3, 4] haveproposed a performance metric computed over the “ROC hypersurface” and (1) is used to justifysuch a metric in a binary classification setting.
Using (1), we can confirm a simple geometric fact whose proof can be found in Appendix A:
Proposition 1. >
ROC( t ) ∈ [ √ 2 , 2] , for all t .
This result reflects the geometric observation that any monotone curve (such as ROC curve) startsand ends at two opposite corners of the ROC space [0 , 1] 2 has an arc length between √ 2 and 2 .
f -divergences Arising from ROC Arc Length3.1 f -divergence of Score Distributions
Among many discrepancies measures, f -divergence has been widely used in many applications.
Definition 1.
Let p and q be densities of two continuous distributions.
An f -divergence is defined as:
D g
[Equation]
p | q ) :
=
E q
[Equation]
g (cid:16) p
[Equation]
z
[Equation]
q
[Equation]
z ) (cid:17)(cid:105) , where g is convex and lower-semicontinuous satisfying g (1) = 0 .
Now let us slightly rewrite (1).
Assuming f − is strictly positive (in which case F − is strictlyincreasing), we can write
[Equation]
(2) where g ( s ) = √ s 2 + 1 −√ 2 .
Equation (2) yields the first important result of this paper: >
ROC( t ) −√ 2is an f -divergence between score densities f
+ and f − since g ( s ) is a convex function and g (1) = 0 .
This result confirms, for any given t , > ROC( t ) is a good discrepancy for measuring positive andnegative scores (i.e., score distributions).
It also explains why the ROC arc length in Figure 1 is agood discrepancy measure: Given t ( x ) = x , the score distributions are same as the data distributionsin each plot.
The arc length of ROCs in Figure 1 plots are f -divergences of the score distributionshence are also f -divergences of data distributions.
Although > ROC( t ) − √ 2 is an f -divergence of score distributions, it is not an f -divergence of thepositive and negative data distributions for general t .
The choice t ( x ) = x in the toy example doesnot have simple analogues for higher dimensional datasets.
Are there choices of t such that the arclengths of their ROC curves are good discrepancy measures on data distribution?
In what follows, weshow when t is a bijective transform of the likelihood ratio p + ( x ) p − ( x ) , > ROC( t ) encodes the differencesbetween p + ( x ) and p − ( x ) in the form of an f -divergence between p + ( x ) and p − ( x ) .
3.2 f -divergences of Data Distributions
Using the law of the unconscious statistician, we can express > ROC( t ) in terms of an expectationwith respect to the negative data density p − ( x ) :
E p − (cid:114)(cid:104) f
+ ( t ( x ))
f − ( t ( x ))
(cid:105) 2 + 1 .
Consider a special family ofscore functions: t ∗ ( x ) = γ (cid:16) p + ( x ) p − ( x ) (cid:17) where γ is any strictly increasing function.
Due to the NeymanPearson lemma [24], ROC( t ∗ ) has the highest TPR at any FPR level.
Geometrically speaking, theydominate all other ROC cuves in an ROC plot.
Hence, we refer to t ∗ as the optimal score and
ROC( t ∗ ) as the optimal ROC curve.
For convenience, we denote the ROC( t ∗ ) as ROC ∗ whichreads “rock star”.
It can be shown that
[Equation]
where the second equality holds due to (cid:82) D a ( x )d x (cid:82)
D b ( x )d x = γ − 1 ( C ) , when γ (cid:16) a ( x ) b ( x ) (cid:17) ≡ C, ∀ x ∈ D .
Whenγ ( x ) =
x , (3) expresses a known result [5], and is often given in plain English as “the density ratioof the likelihood ratio score is the likelihood ratio itself”.
Finally, > ROC ∗ takes an elegant form free from t ∗ or γ : E p − ( x ) (cid:114)(cid:104) p
[Equation]
2 + 1 .
Equivalently,
[Equation]
We can see that the same f -divergence arises from computing the arc length of ROC ∗ .
However,unlike the f -divergence given in (2), (4) is an f -divergence between data distributions, not scoredistributions.
It shows that as long as we use the the optimal scores, the arc length of the optimal
ROC can indeed reflect the differences between data distributions.
From now on, we will refer to 125 >
ROC ∗ − √ 2 as the ROC divergence.
To the best of our knowledge, (4) has not been presented inliterature before.
By definition, the ROC divergence is symmetric.
Moreover, as a result of Proposition 1, the ROCdivergence is upper bounded by 2 − √ 2 and lower bounded by 0.
As we mentioned earlier, ROC ∗has the maximal AUC.
Some geometric properties of ROC ∗ can be found in Section B.
Estimating the Arc Length of ROC ∗4.1 A Variational Objective
To numerically approximate the arc length using samples alone, we leverage that > ROC ∗ − √ 2 is anf -divergence.
Utilizing Fenchel’s duality [18], Authors in [26] show that an f -divergence D g ( p + | p − )has a variational representation:
[Equation]
where g ′ is the convex conjugate of g and the supremum is taken over all measurable functions.
In the case of the ROC divergence, g ( z ) = √ z 2 + 1 − √ 2 with z ∈ [0 , ∞ ] thus g has a convexconjugate g ′ ( z ′ )
= − √ 1 − z ′ 2 − √ 2 , z ′ ∈ [0 , 1] .
Rewriting >
ROC ∗ − √ 2 using the above variationalrepresentation and dropping the −√ 2 , we obtain:
[Equation]
We reparameterize u ( x ) = sin[ v ( x )] , where v ∈ [0 , π/ 2] :
[Equation]
Differentiating the objective in (5) for v and set to zero, we can see the supremum is attained atv ∗ =
atan p + p − .
In other words, the optimal v ∗ is the arctangent likelihood ratio function.
It is also interesting to see how v ∗ is visualized in the ROC plot.
We can see the tangent of ROC ∗at an FPR level s 0 ∈ [0 , 1] is
[Equation]
) , where x 0 is any point in Xthat satisfies the equality γ (cid:16) p + ( x 0 ) p − ( x 0 ) (cid:17) =
˜ F − 1 − ( s 0 ) .
It can be seen that the likelihood ratio characterizesthe slope of ROC ∗ and thus v ∗ =
atan p + p − is the slope angle of ROC ∗ .
Moreover, using (5), we can obtain a relationship between > ROC ∗ and the total variation distancebetween P + and P − (denoted as TV( P + , P − ) ).
Proposition 2.
TV( P + , P − ) has a lower and upper bound expressed via > ROC ∗ :
[Equation]
150 Proof of this proposition can be found in Section C.
This proposition justifies that > ROC ∗ is a validmeasure of the discrepancy between P + and P − .
4.2 A Tractable Objective for Estimating atan p + p −
To use (5) in practice, we need to find an appropriate function class F .
We can simply restrict v to abounded (parametric/non-parametric) function class F and solve the sample version of (5):
[Equation]
where, for ease, we write (cid:80) n + i =1 f ( x + i ) as (cid:80) n + i =1 f ( x i ) , a convention which will adopted henceforth.
In practice, enforcing the boundedness v ∈ [0 , π/ 2] over X is difficult.
We can relax (6) by onlyenforcing the boundedness constraint of v on our finite sample dataset X + ∪ X − .
For example, by letting F be a Reproducing Kernel Hilbert Space (RKHS) [30], we can translate (6)into the following optimization problem:
[Equation]
(7) where H is a RKHS with a positive definite kernel
k
( x , x ′ )
= ⟨ φ ( x ) , φ ( x ′ ) ⟩ , ∥ · ∥ H is the RKHSnorm and λ 2 ∥ v ∥ 2 H is the regularization term.
The optimizer ⟨ ˆ v, φ ( x ) ⟩ is an estimation of v ∗ ( x ) , thearctangent of the likelihood ratio.
(7) is a strictly convex optimization and thus, if a solution ˆ v exists,it must be unique.
Instead of modelling atan p + p − , we can opt for modelling the log likelihood ratio log p + p − .
However,this modelling choice results in an non-convex optimization thus presents extra challenges in thetheoretical analysis.
Details can be found in Section J.
4.3 Finite Sample Guarantee
We show that the solution of (7), ⟨ ˆ v, φ ( x ) ⟩ converges to the true arctangent likelihood ratio (or itsprojection onto H ) as the number of samples goes to infinity.
Below are a few regularity conditions.
Assumption 1.
There exists a unique v ∗ ∈ H , such that E [ ∇ v ℓ ( v ∗ )]
= 0 and ⟨ v ∗ , φ ( x ) ⟩ ∈ [0 , π/ 2]holds for all x ∈ X .
A sufficient condition of the above condition is specified in the following proposition.
Proposition 3.
If there exists a unique v ∗ ∈ H , such that ⟨ v ∗ , φ ( x ) ⟩ = atan (cid:104) p + ( x ) p − ( x ) (cid:105) then Assumption 1 holds.
Proof of this Proposition can be found in Appendix F. Proposition 3 states if model is correctlyspecified and identifiable then Assumption 1 holds.
It is possible that there exists a v ∗ ∈ H satisfying 177 E [ ∇ v ℓ ( v ∗ )]
= 0 which does not meet the boundedness constraint [0 , π/ 2] .
In this paper we onlyconsider situations where Assumption 1 holds, which includes all situations where the model iscorrectly specified and some situations where the model is misspecified.
Assumption 2.
Let n min =
min( n + , n − ) .
There exists a subspace H ∗ := { v ∈ H|∥ v − v ∗ ∥ 2 H ≤δ 2 n min } , such that ∀ v ∈ H ∗ , ∀ x ∈ X + ∪ X − , ⟨ v, φ ( x ) ⟩ ∈ (0 , π 2 ) holds with high probability.
Thesequence δ n min is monotonically decreasing as n min grows to infinity.
Assumption 2 states all v within a vicinity of v ∗ are in the interior of (7)’s feasible region with highprobability.
The following proposition gives a sufficient condition under which Assumption 2 holds.
Proposition 4.
Suppose ∥ φ ( x ) ∥ H ≤ 1 .
If our model is correctly specified as described in Proposition3 and ∀ x ∈ X , atan (cid:104) p + ( x ) p − ( x ) (cid:105) ∈
[ R 1 , R 2 ] , for some R 1 and R 2 such that π 2 > R 2 > R 1 > 0 , thenthere exists N > 0 such that Assumption 2 holds when n min > N .
The proof can be found in Appendix D. Since we use RKHS as the estimator function class, our finalassumption is that v ∗ should be reasonably smooth.
In previous works such an assumption dependson the decay of the integral operator’s eigenvalues [33, 8].
In this paper, we measure the smoothnessusing the range space technique which has recently been adopted in [9, 32].
We defineΣ v :=
[Equation]
[Equation]
where ⊗ denotes the outer product.
Given v 0 ∈ H , Σ v 0 is an integral operator on u ∈ H andΣ
[Equation]
By definition Σ v 0 is a positive, self-adjoint operator, in the sense that ⟨ u, Σ v 0 u ⟩ ≥ 0 , ⟨ u, Σ v 0 v ⟩ =⟨ Σ v 0 , u, v ⟩ , ∀ v, u ∈ H .
Moreover, some algebra shows that Σ v 0 is also a bounded and compactoperator.
See Section G for more details.
Next, we assume the true arctangent ratio function (or its projection) is in the range space of Σ v ∗ .
Assumption 3.
Let R ( Σ v ∗ ) denote the range space of Σ v ∗ .
There exists 0 < β ≤ 1 , v ∗ ∈ R ( Σ βv ∗ ) ,where C β is the fraction power of a compact, positive and self-adjoint operator C .
Note that the larger β is, the smoother the functions in the range space are.
More discussions on therange space assumption can be found in Section 4.2, [32].
Now we are ready to state our theorem:
Theorem 1 (Convergence Rate of ˆ v ) .
Suppose Assumptions 1, 2 and 3 hold and ˆ v exists.
If ∥ φ ( x ) ∥
H ≤ 1 and λ =
[Equation]
4 max( B + [Equation] B − ) , where B + =∥ ( Σ v ∗ + λ I ) − 1 E p + [ φ ( x )]
∥ H , B − =
∥ ( Σ v ∗ + λ I ) − 1 E p − [ φ ( x )] ∥ H , and T ≥ 1 , K > 0 areconstants that do not depend on n min , then there exists a constant N > 0 such that when n min > N ,∥ ˆ v − v ∗ ∥ H
= O p ( n − β/ 4 min ) .
The proof can be found at Appendix E. Theorem 1 shows that, under mild assumptions, ˆ v is indeed agood estimator for atan (cid:104) p + ( x ) p − ( x ) (cid:105) .
Some discussions comparing our results with convergence resultsproved by Nguyen et al.
[25] can be found in Section L. Since atan (cid:104) p + ( x ) p − ( x ) (cid:105) is an optimal score, ourestimator may have some interesting applications such as outlier detection [17] or Neyman-Pearsonclassification [34].
We will defer discussions on those applications in future works.
In the nextsection, we employ our arctangent likelihood ratio estimator in an application of lower bounding themaximal AUC.  Lower Bounding the Maximal AUC
Finding a score function t that approximately maximizes AUC is an important task in binary classification.
Due to the Neyman-Pearson lemma, ROC ∗ has the maximum AUC among all ROC curves.
1.2 FPR0.
T P R 0.
1 FPR 0.
T P R 0.
1 FPR 0.
T P R
Let us denote the AUC of ROC ∗ as AUC ∗ .
Consider the following inequalities:
[Equation]
where L ( a, b ) is a continuous and concave lower bound of the indicator function 1 ( a > b ) .
It can be seen that E p − E p + [ 1 ( t ( x + ) ≥ t ( x − ))]
= (cid:82) [0 , 1] ˜ F + ( ˜ F − 1 − ( s ))
d s is the AUC of ROC( t ) .
Due to the
Neyman-Pearson lemma, the supremum of (i) is only attained when t ( x ) = γ (cid:16) p + ( x ) p − ( x ) (cid:17) where γ is astrictly increasing function.
Replacing the expectations in (ii) with sample averages, we obtain theoptimization problem of AUC maximization [2, 10].
Therefore, we can see that AUC maximizationis a procedure that approximates an optimal score function by maximizing a lower bound of AUC ∗ .
Now, we show a different way of lower bounding AUC ∗ with the help of atan p + ( x ) p − ( x ) .
We have seenthat how ( ˜ F − ( τ ) , ˜ F + ( τ )) parameterizes an ROC curve given a score function t in Section 2.3.
Infact, the surface between ROC ∗ and the diagonal line between (0 , 0) and (1 , 1) can be similarlyparameterized by considering ROC curves of mixtures of positive and negative score distributions.
Let F ∗ + and F ∗− denote CDFs of any optimal score.
For α ∈ [0 , .
5] , we define FPR and TPR forα -mixtures of F ∗ + and F ∗− :
[Equation]
We can see the 2-dimensional coordinate r ( τ, α ) :
= ( ˜ F ∗− ( τ, α ) , ˜ F ∗ + ( τ, α )) parameterizes the areabetween ROC ∗ and the diagonal line in [0 , 1] 2 : When fixing α and varying τ , the coordinates giverise to a smooth curve in ROC space from [0 , 0] to [1 , 1] .
When α = 0 , such a curve is ROC ∗ .
Whenα = . 5 , such a curve is the diagonal line.
When fixing τ = τ 0 and varying α , the coordinates producea straight line segment connecting ( ˜ F ∗− ( τ 0 , 0) , ˜ F ∗
+ ( τ 0 , 0)) and ( ˜ F ∗− ( τ 0 , . 5) , ˜ F ∗ + ( τ 0 , . 5)) .
The left plotin Figure 2 visualizes this parameterization.
Now the surface area sandwiched between ROC ∗ andthe diagonal line can be computed using a surface integration:
[Equation]
where × denotes the cross product.
After some algebra and applying the Fenchel duality technique in
Section 4.1, we prove that AUC ∗ can be expressed as the supremum of a variational objective similarto (5):
Proposition 5.
AUC
[Equation]
[Equation]
where w ( τ ) := sin( τ + π 4 ) · | F ∗ + ( τ ) − F ∗− ( τ ) | .
The supremum of (10) is attained at v ∗ =
atan p + p − .
The proof can be found in Appendix H in the supplementary material.
A lower bound of A can beobtained by restricting v to a function class.
Evaluating w requires us to evaluate atan p + ( x ) p − ( x ) , F ∗ + and F ∗− which are not readily available.
However,
Section 4.3 shows that the empirical estimator (7) is a consistent estimator of atan p + ( x ) p − ( x ) under mildconditions.
Therefore, we propose the following two-step procedure to approximately lowerbound A :
Obtain ˆ t ( x ) := ⟨ ˆ v, ϕ ( x ) ⟩ using (7).
Approximate F ∗ + and F ∗− using ˆ F + and ˆ F − which areempirical CDFs of ˆ t ( x + ) and ˆ t ( x − ) .
Optimize the empirical version of (10) by restricting v to a feasible function class and plugging in estimates obtained in the earlier step, i e.,
[Equation]
where ˆ w ( τ ) := sin( τ + π 4 ) (cid:12)(cid:12)(cid:12) ˆ F + ( τ ) − ˆ F − ( τ ) (cid:12)(cid:12)(cid:12) .
Note that (11) is nothing but a weighted sample objective (6).
Thus, it can be easily optimized by thesame algorithm that solves a weighted version of (6) given the approximated weights in the first step.
In practice, we simply run (7) twice:
The first time we run it without weights then run it again withweights ˆ w (cid:2)
ˆ t ( x i ) (cid:3) calculated from the first run.
Since the above algorithm also approximates an optimal score ( atan p + p − ) by maximizing a lowerbound of AUC ∗ , it is natural to wonder how the maximizer ˆˆ v of (11) would perform in AUCmaximization tasks.
In the next section, we show our two-step algorithm achieves a promising AUCperformance when compared with an AUC maximizer.
Comparing to the traditional AUC maximizer, the proposed algorithm has a lower computational costin the offline setting.
Without loss of generality, assume n + = n − = n .
The AUC maximizationderived from the Wilcoxon-Mann-Whitney statistic [14] has a computational complexity O ( n 2 ) .
In comparison, the objective (11) is a summation of sin / cos( v ( x )) evaluated at each datum, socomputing the objective/gradient has a computational complexity O ( n ) .
Computing ˆ F + and ˆ F −requires sorting our dataset which has an average complexity O ( n log n ) .
However, once our datasetsare sorted, ˆ F + (cid:0) ˆ t ( x 0 ) (cid:1) =
in + , where i is the index of ˆ t ( x 0 ) in the sorted set { ˆ t ( x i ) } n + i =1 .
We validatethis statement by plotting the wall clock time in Figure 4.
See Section I for details.
The two-stepprocedure’s computation time grows at a much slower rate than the offline AUC maximization via apairwise loss function. Note that when the surrogate loss L is the squared loss and t ( x ) is restrictedto a linear model, the computation of the pairwise loss can be reduced to O ( n ) [36]. However, thisnice result does not apply to more general choices of L and models.(9) computes AUC ∗ differently from the conventional Wilcoxon-Mann-Whitney statistic: Our approach divides the space sandwiched between ROC ∗ and the diagonal into small surface elementsthen adds them up while Wilcoxon-Mann-Whitney statistic adds up all histogram bars which are
TPRs at different FPR levels.
Our approach require ˜ F + and ˜ F − to be differentiable with respect toτ , which means the score distributions cannot be discrete.
However, Wilcoxon-Mann-Whitney cancompute the AUC of discrete score distributions without any problem.
This difference is visualizedin the middle and right plot of Figure 2.
Numerical Comparison of Divergences and TV Bounds
In this experiment, we numerically compare the ROC divergence, the upper and lower bound in
Proposition 2 with several other divergences and some known bounds of TV in Figure 3.
In thisnumerical simulation, p + = N (0 , 1) and p − = N ( δ, 1) .
We plot ROC divergence, JS divergence,
Wasserstein distance and TV between p + and p − as δ grows from 0 to 5.
We can see the (rescaled)
ROC divergence closely resembles TV .
When δ > 1 . 45 , the upper bound given in Proposition 2 isthe tightest among known TV upper bounds (Pinsker’s upperbound, Bretagnolle & Huber’s upperbound, Le Cam’s upper bound).
This suggests that combining our upperbound with existing boundsmay produce an even tighter bound for TV .
Imbalanced Classification on CIFAR-10
In this section, we test if the ˆˆ v obtained in our two-step procedure (11) is indeed a good score
functionin terms of AUC in imbalanced classification tasks.
We use a widely known image classificationdataset CIFAR-10 [19].
The performance is compared with an AUC maximizer which maximizesthe empirical lower bound in (8) and a vanilla logistic regression classifier.
We set the surrogateloss L ( a, b ) :
= − (1 − ( a − b )) 2 in the AUC maximizer, as suggested in [10].
All methods uselinear models with no regularization terms since our models are simple and samples are sufficient.
Particularly, the ˆ t and ˆˆ v in our two-step algorithm is obtained using (7) by setting ϕ ( x ) = x andλ = 0 and the AUC maximization (AUC max) is implemented using SPAUC method [21].
Instead of using the raw features, we extract 50 dimensional bounded features by training a residualnetwork [15] on the training dataset using the 10-class cross entropy loss.
The structure of the networkis included in the supplementary material.
After obtaining features, we construct datasets for 10different one-versus-the-rest classification tasks.
For a single task, we pick a class and attain X + byrandomly sampling from this class n + times in the training set.
Similarly, X − is attained by randomlysampling from the rest of the classes n − times.
In our experiments, we set n + = 24 , 48 , 72 , 96 , 120and fix n − = 1000 to create imbalanced positive and negative datasets.
We run all three methods andobtain the corresponding score functions.
For each class, we repeat the experiment 96 times usingdifferent random samples.
We use the testing and training split provided by the dataset itself.
Our experiments can be seen as a transfer learning task which reuses predictive features trained for amulti-class classifier for one-versus-the-rest binary classification tasks.
The average AUCs computed on the testing dataset and their standard errors over 96 runs overdifferent n + sample sizes are shown in Figure 5.
It can be seen that our method has approximatelyequal performance with the AUC maximizer despite not directly maximizing the AUC and having alower computational complexity.
This observation indicates that ˆˆ v can be a good score function in
AUC maximization tasks.
Both of the methods significantly outperform vanilla logistic regression.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?[Yes]
In introduction, we discussed the background/usages ROC curves, f -divergenceestimation and provided a full paragraph summary of the contribution of this paper.
Wegive the same contribution rundown in the abstract as well.
(b) Did you describe the limitations of your work? Yes
] In Section 5, we included a brief discussion comparing the proposed maximal
AUC lower
bounding
approach vs the classic Wilcoxon-Mann-Whitney statistic
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Full instructions on how to reproduce our experiments is provided in the supplementary material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 7(c)
Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?[N/A
The Computation time is not compared in this paper.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section
(b) Did you mention the license of the assets?
[No] The authors of CIFAR-10 do not specify a license for the dataset.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] We did not create any new asset in this research.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] We did not collect any data ourselves.
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A] CIFAR-10 dataset we use in our experiment does not contain any identifiable
human information.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Proof of Proposition 1
Proof.
Jensen’s inequality:
[Equation]
Here we prove a result regarding some other geometric properties of ROC ∗ .
Proposition 6.
ROC ∗ is a convex curve and > ROC ∗ is the longest among all convex ROC curves.
Proof.
First, we show ROC ∗ is a convex curve.
To show ROC ∗ is convex, we only need to show˜ F + ( ˜ F − 1 − ( s )) is a concave function.
This can be verified by checking the sign of ∂ 2 s ˜ F + ( ˜ F − 1 − ( s )) :
[Equation]
where the second equality is due to (3) and x 0 is any point in X that satisfies the equalityγ (cid:16) p + ( x 0 ) p − ( x 0 ) (cid:17)
=
˜ F − 1 − ( s ) .
Further, we can show that,
[Equation]
Since γ is a strictly monotone increasing function, the first factor is non-negative and the second factor is also strictly positive due to the our assumption on the positivity of f − .
We have ∂ 2 s F + ( F − 1 − ( s ; t ∗ )) ≤ 0 .
Moreover, at any FPR level s ∈ [0 , 1] , the Neyman-Pearson lemma [24] implies
[Equation]
where ˜ F ′ + and ˜ F ′− are TPR and FPR of any other score function.
In words, ROC ∗ dominates allother ROC curves.
Since ROC ∗ is convex and encloses all other ROC curves, our claim follows
Archimedes’s Second Axiom: among all convex curves with the same endpoints, the one encloses allother curves has the longest arc length.
C Proof of Proposition 2
Proof.
Using the integral probability metric representation of TV( P + , P − ) [31], we can write:
[Equation]
Some algebra can show that z ≥ sin( z )
a
[Equation]
and − z ≥ [Equation] − arcsin
[Equation]
a
[Equation]
afor all a ∈ [0 , 1] and z ∈ [0 , π/ 2] .
Therefore
[Equation]
[Table]
Similarly, multiplying both sides of the second equality above by 2 π , we obtain
[Equation]
D Proof of Proposition 4
Proof.
∀ v ∈ H ∗ , |⟨ v, φ ( x ) ⟩−⟨ v ∗ , φ ( x ) ⟩| ≤ ∥ v − v ∗ ∥ H ∥
φ ( x ) ∥
H ≤ δ n min .
If δ n min < min( R 1 , π 2 −
R 2 ) then ⟨ v, φ ( x ) ⟩ ∈ (0 , π 2 ) holds uniformly for every x ∈ X .
As δ n min is a decaying sequence,there always exists an N such that δ n min ≤ min( R 1 , π 2 − R 2 ) holds for n min ≥ N .
E Proof of Theorem 1
To reduce the visual clutter, in this section, ∥ v ∥ represents the Hilbert space norm of v , defined as 479 (cid:112) ⟨ v, v ⟩ .
We simplify E p + [ v ( x )] as E + [ v ( x )] whenever it does not lead to confusion.
Proof.
Define H ∗ := { v ∈ H|∥
v − v ∗ ∥ 2 ≤ δ 2 } .
Consider an optimization that is similar to (7):
[Equation]
Define ˜ u :
=
˜ v − v ∗ and we have the following equality due to the KKT conditions of (12)
[Equation]
where ν is a Lagrangian multiplier and ν ≥ 0 .
Multiplying both sides by ˜ s
= ( Σ v ∗ + λ I ) − 1 ˜ u , wehave
[Equation]
Let g ( v ) :
= ⟨ ˜ s, ∇ v ℓ ( v ) + λv + 2 ν ( v − v ∗ ) ⟩ , we can applying Mean Value Theorem (MVT) on thescalar valued function g ( v ) :
[Equation]
13) where ¯ v =
av ∗ + (1 − a )
ˆ v for some a ∈ [0 , 1] .
Knowing g
(˜ v ) =
and g ( v ∗ ) = ⟨ ˜ s, ∇ v ℓ ( v ∗ )
+ λv ∗ ⟩ ,we can translate (13) into
[Equation]
where I is the identify matrix.
Focusing on the RHS, we have
[Equation]
[Equation]
The first line is due to the fact that ⟨ 2 ν ˜ s, ˜ u ⟩ ≥ 0 .
Use the inequality (15) on (14), we get the inequality
[Equation]
[Equation]
First, let us inspect a .
Using MVT on sin ⟨ v, φ ( x ) ⟩ , v ∈ H ∗ and applying Hölder’s inequality, we getsin
[Equation]
[Equation] Define Σ + v := E + [sin ⟨ v, φ ( x ) ⟩ φ ( x ) ⊗ φ ( x )] and ˆ Σ
+ v as its empirical counterpart approximatedusing X + .
We can see that a = ⟨ ˜ s, [ Σ + v ∗ − Σ +¯ v ]˜ u ⟩ + ⟨ ˜ s, [ Σ − v ∗ − Σ − ¯ v ]˜ u ⟩ .
Moreover,
[Equation]
( i ) is due to (17).
Following a similar line of reasoning, we can see
[Equation]
By setting δ ≤ 4 max (cid:16) ∥ ( Σ v ∗ + λ I ) − 1 E + [ φ ( x )] ∥ , ∥ ( Σ v ∗ + λ I ) − 1 E − [ φ ( x )] ∥ (cid:17) − 1 , we have
[Equation]
Now we inspect b .
We can see | b | ≤ (cid:12)(cid:12)(cid:12) ˜ s ˆ [Equation] + ¯ v ˜ u − E
+
[Equation]
˜
[Equation]
ˆ Σ + ¯ v ˜ u
[Equation]
+
[Equation]
˜
[Equation]
ˆ
[Equation]
− ¯ v ˜ u
[Equation]
ˆ
[Equation]
− ¯
[Equation]
˜ u (cid:105)(cid:12)(cid:12)(cid:12) .
Define ascalar random variable
[Equation]
By definition 1 n + (cid:80) n + i =1 Z ( i ) f
= ˜ s ⊤ ˆ Σ + f ˜ u.
Therefore
[Equation]
n +
[Equation]
Since 0 ≤ Z ( i ) v ≤ ∥ ˜ s ∥ · ∥ ˜ u ∥ · ∥ φ ( x ) ∥ 2 ≤ ∥ ( Σ v ∗ + λ I ) − 1 ˜ u ∥∥ ˜ u
∥ ≤ ∥ ˜ u
∥ 2 λ , using Uniform Law of
Large Number for bounded random variable (Theorem 4.10, [35]),
[Equation]
n [Equation] with high probability, where R n + ( F Z ) is the Rademacher complexity of the function class of Z v .
Itremains to bound R n + ( F Z ) .
It can be seen that Z f = h [ ⟨ v, φ ( x ) ⟩ ] where h is a Lipschitz continuosfunction with Lipschitz constant ∥ ˜ u
∥ 2 λ .
Hence, due to Ledoux–Talagrand contraction inequality (see,e.g., (5.61) in [35]), R n + ( F Z ) is upperbounded by,
[Equation]
where C 0 is a universal constant.
The last inequality is due to Corollary 14.5 in [35].
Therefore
[Equation]
and similarly,
[Equation]
Therefore,
[Equation]
19) with high probability.
Substituting (18)and (19) into (16), we get
[Equation]
Using triangle inequality and Hölder’s inequality, we have
[Equation]
Due to Assumption 1, E [ ∇ v ℓ ( v ∗ )]
= 0 .
Hence,
[Equation]
We can see
[Equation]
holds with high probability and C 1 is a universal constant (due to Lemma 2).
Moreover, since v ∗ ∈ R ( Σ βv ∗ ) , there exists g ∈ H , v ∗ =
Σ βv ∗ g .
Notice Σ v ∗ is a bounded,compact, self-adjoint linear operator (see Section G).
Therefore, Hilbert-Schmidt Theorem indicates,Σ v ∗ =
(cid:80)
i α
i ψ i ⟨ ψ i , ·⟩ , where ψ i , α
i are eigenfunctions and eigenvalues of Σ v ∗ respectively.
Hence,∥
[Equation]
Combine (20), (21) and (22) and cancel ∥ ˜ u ∥ , we can conclude that
[Equation]
with high probability.
Set λ = max( C 1 ,C 0 , 1) n 1 /min , we have
[Equation]
holds with high probability.
Therefore, ∃ N 2 , when n min > N 2 , ∥ ˜ u
∥ =
O p ( n − β/ 4 min ) .
Since ∥ ˜ u ∥ = o p (1) , as long as δ ≥ K · n − β/ 4 min where K > 0 is a constant, there exists a constant Nsuch that, when n min > N , ˜ v is in the interior of H ∗ with high probability.
When this happens, theconstraint v ∈ H ∗ is no longer active.
This means ˜ v is a stationary point of the objective function in(12) .
Moreover, ˜ v ∈ H ∗ , so it is in the feasible region of (7) thanks to Assumption 2.
This furtherindicates that ˜ v is also a solution to (7).
As (7) is a strictly convex optimization problem, ˜ v is also itsonly solution.
Therefore ˜ v =
ˆ v and ∥
ˆ v − v ∗ ∥ =
∥
˜ v − v ∗ ∥
=
O p ( n − β/ 4 min ) .
Lemma 2.
Given any v ∗ ∈ H such that E [ ∇ v ℓ ( v ∗ )]
= 0 , if ∥ φ ( x ) ∥
H ≤ B then
P ( ∥∇ v ℓ ( v ∗ ) ∥ H > δ ) ≤ 4 exp (cid:18) − n min δ B(cid:19) .
Proof.
Write down the definition of ∇ v ℓ ( v ∗ ) .
Notice∇ v ℓ ( v ∗ ) = − 1 n + n + (cid:88) i = cos ⟨ v, φ ( x i ) ⟩ φ ( x i ) (cid:124) (cid:123)(cid:122) (cid:125) a +n − n − (cid:88)
i = sin ⟨ v, φ ( x i ) ⟩ φ ( x i ) (cid:124) (cid:123)(cid:122) (cid:125) b .
By using Hilbert-space Hoeffding’s inequality [29], we know for all δ a , δ b >
P ( ∥ a − E
[ a ] ∥ H > δ a ) ≤ 2 exp (cid:18) − Cn + δ 2 a
B(cid:19) and P ( ∥ b − E [ b ] ∥ H > δ b ) ≤ 2 exp (cid:18) − Cn − δ 2 b
B(cid:19) , where C is a constant.
Let δ =
δ a = δ b ,
[Equation]
where the first equality used the condition that E [ ∇ v ℓ ( v ∗ )]
=
E [ a ] + E
b ] = 0 .
This completes theproof.
F Proof of Proposition 3
Proof.
We start from the definition of E [ ∇ v ℓ ( v ∗ )] :
[Equation]
where the third equality is due to the fact that ⟨ v ∗ φ ( x ) ⟩ = atan p + ( x ) p − ( x ) .
Since p + /p − ∈ [0 , ∞ ) ,⟨ v, φ ( x ) ⟩ ∈ [0 , π/ 2) .
As v ∗ is unique by assumption, Assumption 1 holds.
G Properties of Operator Σ v 0
By construction, it is easy to verify that Σ v 0 is self-adjoint.
First, we prove that the integral operatorΣ
[Equation]
[Equation]
[Equation]
[Equation]
+ E − [cos
[Equation]
, φ
[Equation] x ) ⟩ [Equation] ( x ) · u ( x )] , is a bounded operator.
For all u ∈ Ball(0 , 1) , where Ball(0 , 1) is the unit ball in ∥ · ∥ H ,∥ Σ [Equation] cos
[Equation]
0 , φ ( x )
[Equation]
( x ) · u ( x )]
∥ H
[Equation]
[Equation]
[Equation]
[Equation] [Equation] ( x ) ∥ H ] . Hence, Σ v 0 is a bounded operator as long as ∥ φ ( x ) ∥ H is bounded.
Second, we show Σ v 0 is trace class hence compact.
Let ψ i , i ∈ N be an orthonormal basis in H , then(cid:88) i ⟨
[Equation] , Σ [Equation] 0 ψ
[Equation]
⟩ = E + [sin ⟨ v 0 , φ ( x ) ⟩ (cid:88) i ∈ N
[Equation]
ψ [Equation] , φ ( x ) ⊗ φ ( x ) ψ i ⟩ ] + E − [cos ⟨ v 0 , φ ( x ) ⟩ (cid:88) i ∈ N ⟨ ψ i , φ ( x ) ⊗ φ ( x ) ψ i ⟩ ] =
E + [sin ⟨ v 0 , φ ( x ) ⟩ (cid:88) i ∈ N ⟨ ψ i , φ ( x ) ⟩ 2 ] + E − [cos ⟨ v 0 , φ ( x ) ⟩ (cid:88) i ∈ N ⟨ ψ i , φ ( x ) ⟩ 2 ]
= E + [sin ⟨ v 0 , φ ( x ) ⟩ · ∥ φ ( x ) ∥ 2 H ] + E − [cos ⟨ v 0 , φ ( x ) ⟩ · ∥ φ ( x ) ∥ 2 H ] < ∞ holds as long as ∥ φ ( x ) ∥ H is bounded.
This shows Σ v 0 is trace-class and therefore, compact.
H Proof of Proposition 5
Proof.
Let us define for α ∈ [0 , .
[Equation]
We can see that r ( τ, α ) :
= ( ˜ F ∗− ( τ, α ) , ˜ F ∗ + ( τ, α )) is a parameterization for the space between ROC ∗and the diagonal from (0 , 0) to (1 , 1) .
We can compute the surface area using the surface integralformula:
[Equation]
where ∂ τ r ( τ, α ) =  
∂ τ ˜ F ∗− ( τ, α ) ∂ τ ˜ F ∗ + ( τ, α )   and ∂ α r ( τ, α ) =  
∂ α ˜ F ∗− ( τ, α ) ∂ α ˜ F ∗ + ( τ, α )   .
It can be seen that∂ α r
( τ, α ) =
(cid:34) F ∗− ( τ ) − F ∗ + ( τ )
F ∗ + ( τ ) − F ∗− ( τ ) (cid:35) for all α .
Rewrite A 0 :
A 0 = (cid:90) dom( τ ) (cid:90) [0 ,.
[Equation]
23) where a ( τ ) =
[Equation]
F
[Equation]
and b ( τ ) =
[Equation]
∂
[Equation]
F
[Equation]
Both a and b are free from α .
Rewritingthe cross product in (23) in a different form, we obtain
[Equation]
where θ ( τ ) is the angle between a ( τ ) and b ( τ ) .
b ( τ ) is the tangent vector of the ROC ∗ .
Knowingthe slope of ROC ∗ is the likelihood ratio (see Section 4.1) and a ( τ ) points at the 45 degree downwardregardless of τ , we can see θ ( τ ) =
(cid:104) atan p ( x ) q ( x ) (cid:105)
+ π 4 .
Using the fact that f ∗ + ( t ( x ))
f ∗− ( t ( x ))
=
p + ( x ) p − ( x ) and thelaw of unconscious statistician:
[Equation]
Replacing (cid:114) p + ( x ) p − ( x ) 2 + 1 with its Fenchel dual as introduced in Section 4.1 and pulling the sup out ofthe expectation yields the desired result.
Differentiating the objective (10) with respect to v and setting the derivative to zero, we can see thatsuperemum is attained at v ∗ =
atan p + p − .
- -   4 x 0.
[Figure]
I Experiment Setup
In the wall clock time comparison in Section 5, we implement the AUC maximization and two-stepprocedure using MATLAB’s Optimization Toolbox.
Both methods are written in fully vectorized code.
The first order derivatives are provided to the fmincon and fminunc to accelerate the computation.
Code can be found in the supplementary material.
In Section 7, we reduce the dimension of CIFAR-10 dataset to 50.
We first train a residual neuralnetwork [15] using logistic regression on all 10 classes.
This 103-layer network structure wasborrowed from a MATLAB tutorial ( https://www.mathworks.com/help/deeplearning/ug/train-residual-network-for-image-classification.html ).
MATLAB provides a pretrained version of this network.
To obtain bounded features, we append a fully connected linear layer(output dimension 50) and a bounded activation layer (clipped-relu) to the last average pooling layerin the network.
We freeze the earlier layers and only train the last two layers for 5 epochs.
The dataset and the code that reproduces Figure 5 can be found in the supplementary materials.
Weinvite reviewers to reproduce our results.
J Estimating
[Equation]
We can also leverage that v ∗ is the arctangent of the likelihood ratio and introduce mild assumptionson p + and p − .
When p + ( x ) and p − ( x ) are both members of the exponential family and share thesame sufficient statistic h ( x ) ∈ R m , then ∃ v ∗ ∈ R m such that log (cid:104) p + ( x ) p − ( x ) (cid:105) =
⟨ v ∗ , h ( x ) ⟩ + C ,where C is a constant.
If we choose to parameterize the log likelihood ratio using a linear model,⟨ v , h ( x ) ⟩ + v 0 , then (6) becomes(ˆ v , ˆ v 0 ) :
= argmax v ∈ R m , v 0 ∈ R n + n + (cid:88) i = sin[atan exp( ⟨ v , h ( x i ) ⟩ + v 0 )] + 1 n − n − (cid:88) i = cos[atan exp( ⟨ v , h ( x i ) ⟩ + v 0 )] .
Note we do not have to restrict the optimization to a bounded function family as log (cid:104) p + ( x ) p − ( x ) (cid:105) ∈ R ,and ⟨ ˆ v , h ( x ) ⟩
+ ˆ v 0 is an estimate of the likelihood ratio.
In this paper, we focus on (7) since the objective in (25) is non-convex with respect to v thus presentsextra challenges in the theoretical analysis, although (25) is easier to implement in practice due to itsunconstrained nature.
K Numerical Simulation of atan p + p − Estimation
We draw 100 samples from X + and X − and solve (7) to estimate the arctangent density ratio.
Theestimated arctangent density ratio with standard deviation (over 72 runs) are plotted in Figure 6.
Weuse Gaussian kernel and hyperparameters (kernel bandwidth and regularization parameter) are tunedusing cross validation.
We observe that the estimated arctangent ratio using the proposed method is very close to the groundtruth and has a small standard deviation.
L Comparison with Convergence Result in Nguyen et al. [25]
The convergence of (log) density ratio estimation have been developed for two KL divergencebased estimators [25].
However, these convergence theories are not general theories for arbitraryf -divergences.
Thus their proofs cannot be easily applied to our ROC divergence.
Moreover, Theorem 1 is not a minor modification of convergence theories in [25].
Specifically,
Nguyen et al.
[25] prove the likelihood ratio converges in Hellinger distance, while we prove thearctangent likelihood ratio converges in Hilbert space norm.
The proofs rely on completely differentmachinery and assumptions.
These technical results depend on the variational objective functions theestimators maximize and are not interchangeable.