Optimal Binary Classification Beyond Accuracy
Anonymous Author(s)
AffiliationAddress email
Abstract
The vast majority of statistical theory on binary classification characterizes performance in terms of accuracy.
However, accuracy is known in many cases topoorly reflect the practical consequences of classification error, most famouslyin imbalanced binary classification, where data are dominated by samples fromone of two classes.
The first part of this paper derives a novel generalization ofthe Bayes-optimal classifier from accuracy to any performance metric computedfrom the confusion matrix.
Specifically, this result (a) demonstrates that stochasticclassifiers sometimes outperform the best possible deterministic classifier and (b)removes an empirically unverifiable absolute continuity assumption that is poorlyunderstood but pervades existing results.
We then demonstrate how to use this generalized Bayes classifier to obtain regret bounds in terms of the error of estimatingregression functions under uniform loss.
Finally, we use these results to developsome of the first finite-sample statistical guarantees specific to imbalanced binaryclassification.
Specifically, we demonstrate that optimal classification performancedepends on properties of class imbalance, such as a novel notion called Uniform
Class Imbalance, that have not previously been formalized.
We further illustratethese contributions numerically in the case of k -nearest neighbor classification.
Introduction
Many binary classification problems exhibit class imbalance, in which one of the two classes vastlyoutnumbers the other.
Classifiers that perform well with balanced classes routinely fail for imbalancedclasses and developing reliable techniques for classification in the presence of severe class imbalanceremains a challenging area of research
[He and Ma, 2013, Krawczyk, 2016, Fernández et al., 2018].
Many practical approaches have been proposed to improve performance under class imbalance,including reweighting plug-in estimates of class probabilities [Lewis, 1995], resampling data toimprove class imbalance [Chawla et al., 2002] or reformulating classification algorithms to optimizedifferent performance metrics [Dembczynski et al., 2013, Fathony and Kolter, 2019, Joachims, 2005].
Extensive discussion of practical methods for handling class imbalance are surveyed in the books of
He and Ma [2013] and Fernández et al.
Despite the pervasive challenge of class imbalance, our theoretical understanding of class imbalanceis limited.
The vast majority of theoretical performance guarantees for classification characterizeclassification accuracy or, equivalently, misclassification risk [Mohri et al., 2018], which is typicallyan uninformative measure of performance for imbalanced classes.
Under measures that are used withimbalanced classes in practice, such as precision, recall, F β scores, and class-weighted scores [Van Rijsbergen, 1974, 1979], existing theoretical guarantees are limited to statistical consistency, in that thealgorithm under consideration asymptotically optimizes the metric of choice [Koyejo et al., 2014,
Menon et al., 2013, Narasimhan et al., 2014]; specifically, there is no finite-sample theory that wouldallow comparison of an algorithm’s performance to that of other algorithms or to theoretically optimalperformance levels.
Additionally, existing theory for classification does not explicitly model the effects of class imbalance, especially severe imbalance (i.e., as the proportion of samples from the rareclass vanishes), and hence sheds little light on how severe imbalance influences optimal classification.
This paper provides two main contributions.
First, in Section 4, we provide a novel characterization ofclassifiers optimizing general performance metrics that are functions of a classifier’s confusion matrix.
This characterization generalizes a classical result, that the Bayes classifier optimizes classificationaccuracy, to a much larger class of performance measures, including those commonly used inimbalanced classification, while relaxing certain empirically unverifiable distributional assumptionsthat pervade existing such results.
Interestingly, we show that, in general, a Bayes classifier alwaysexists if one considers stochastic classifiers, but not if one considers only deterministic classifiers.
Wethen use this result to provide relative performance guarantees under these more general performancemeasures, in terms of the error of estimating the class probability (regression) function under uniform( L 8 ) loss.
This motivates our second main contribution: an analysis of k -nearest neighbor ( k NN) classificationunder uniform loss.
In doing so, we also propose an explicit model of a sub-type of class imbalance,which we call Uniform Class Imbalance, and we show that the k NN classifier behaves quite differentlyunder Uniform Class Imbalance than under other sub-types of class imbalance.
To the best of ourknowledge, such sub-types of class imbalance have not previously been distinguished in either thetheoretical or practical literature, and we hope that identifying such relevant features of imbalanceddatasets may facilitate development of classifiers that perform well on specific imbalance problems ofpractical importance.
Collectively, these contributions provide finite-sample performance guaranteesfor nonparametric binary classification under performance metrics that are appropriate for imbalanceddata and show how optimal performance depends on the nature of imbalance in the data.
Related Work
Here, we discuss how our results relate to existing theoretical guarantees for imbalanced binaryclassification and prior analyses of k NN methods.
2.1 Theoretical Guarantees for Imbalanced Binary Classification
Statistical learning theory has studied classification extensively in terms of accuracy [Mohri et al.
However, when classes are severely imbalanced, accuracy ceases to be an informative measureof performance [Cortes and Mohri, 2004], necessitating guarantees in terms of other performancemetrics.
Several papers have sought to address this [Narasimhan et al., 2014, 2015, Koyejo et al. ,2014, Yan et al., 2018, Wang et al., 2019a] by generalizing the Bayes optimal classifier, a well-knownclassifier that provably optimizes accuracy, to more general performance measures better reflectingthe desiderata of imbalanced classification. However, existing results make empirically unverifiableassumptions about the distribution of the data, leaving questions about their relevance to real data.
We discuss these assumptions in detail in Section 4, where our main result, Theorem 3, leverages theidea of stochastic classification to relax these assumptions. Another body of closely related theoretical work studies Neyman-Pearson classification, whichattempts to minimize misclassification error on one class subject to constraints on misclassificationerror on other classes, analogous to the approach of statistical hypothesis testing. While substantialtheoretical guarantees do exist for Neyman-Pearson classification [Rigollet and Tong, 2011, Tong,2013, Tong et al., 2016], these focus on performance within the Neyman-Pearson framework, ratherthan under general performance measures as in our work, and we know of no work consideringstochastic classification under the Neyman-Pearson framework. Interestingly, our use of stochasticclassifiers in Theorem 3 parallels classical results in hypothesis testing [Lehmann and Romano, 2006],and our proof of Theorem 3 involves a reduction (Lemma 21 in the Appendix) of optimization ofgeneral classification performance measures to Neyman-Pearson classification.
Meanwhile, many practical approaches to handling class imbalance, such as class-weighting andresampling have been proposed, but the theoretical understanding of these methods is limited.
Class-weighting is a natural choice in applications where costs, or cost ratios [Flach, 2003], can beexplicitly assigned and, in the case of binary classification, is statistically equivalent to thresholdselection, which we discuss later in this paper
[Scott, 2012].
In practice, resampling appears to bethe most popular approach to handling class imbalance
[He and Ma, 2013].
Undersampling the dominant class is straightforward and can provide computational benefits with little loss in statisticalperformance [Fithian and Hastie, 2014], while interest in oversampling rare classes, sometimesreferred to as data augmentation, has grown with the advent of sophisticated generative models toproduce additional data [Mariani et al., 2018].
However, the theoretical ramifications of oversamplingtechniques used for imbalanced classification, most commonly variants of SMOTE [Chawla et al.
,2002], are poorly understood.
2.2 k NN Classification and Regression
The k NN classifier is one of the oldest and most well-studied nonparametric classifiers Fix and
Hodges [1951].
Early theoretical results include, Cover and Hart [1967], who showed that themisclassification risk of the k NN classifier with k “ 1 is at most twice that of the Bayes-optimalclassifier, and Stone [1977], who showed that the k NN classifier is Bayes-consistent if k Ñ 8 andk { n Ñ 0 .
Extensive literature on the accuracy of k NN classification has since developed [Devroyeet al., 1996, Györfi et al., 2002, Samworth, 2012, Chaudhuri and Dasgupta, 2014, Gottlieb et al.
,2014, Biau and Devroye, 2015, Gadat et al., 2016, Döring et al., 2018, Kontorovich and Weiss, 2015,
Gottlieb et al., 2018, Cannings et al., 2019, Hanneke et al., 2020].
Rather than accuracy bounds for k NN classification, the bounds on uniform error we present in
Section 5 are most closely related to risk bounds for k NN regression, of which the results of Biauet al.
[2010] are representative.
Biau et al.
[2010] gives convergence rates for k NN regression in
L 2 risk, weighted by the covariate distribution, in terms of noise variance and covering numbers ofthe covariate space.
While closely related to our bounds on uniform ( L 8 ) risk, their results differ inat least three main ways.
First, minimax rates under L 8 risk are necessarily worse than under L 2risk by a logarithmic factor, as implied by our lower bounds.
Second, the fact that Biau et al.
[2010]use a risk that is weighted by the covariate distribution allows them to avoid our assumption thatthe covariate density is lower bounded away from 0 , whereas, the lower boundedness assumption isunavoidable under L 8 risk.
Finally, Biau et al.
[2010] assume additive noise with finite variance;
Bernoulli noise is crucial for us to model severe class imbalance.
Research on k NN for imbalanced classification has focused on algorithmic modifications, which aresurveyed by Fernández et al.
Examples include prototype selection [Liu and Chawla, 2011,
López et al., 2014, Vluymans et al., 2016], and gravitational methods [Cano et al., 2013, Zhu et al.
We are aware of no statistical guarantees exist for such methods.
Setup and Notation
Let p X , ρ q be a separable metric space, and let Y “ t 0 , 1 u denote the set of classes.
For anyx P X and ϵ ą 0 , B p x, ϵ q : “ t z P X : ρ p x, z q ă ϵ u denotes the open radius-
ϵ ball aroundx .
Consider n independent samples p X 1 , Y 1 q , . .
, p X n , Y n q drawn from a distribution P X,Y on
X ˆ Y with marginals P X and P Y .
For positive sequences t a n u 8 i “ 1 and t b n u 8 i “ 1 , a n — b n means 125 lim inf n Ñ8 a n { b n ą
and lim sup n Ñ8 a n { b n ă 8 .
To optimize general performance metrics, we must consider stochastic classifiers.
Formally, letting
B : “ t Y „ Bernoulli p
p
q : p P r 0 , 1 su denote the set of binary random variables, a stochasticclassifier can be modeled as a mapping p Y :
X Ñ B , where, for any x P X , E r p
Y
p x qs is the probabilitythat the classifier assigns x to class 1 .
We use SC to denote the class of stochastic classifiers.
The true regression function η ˚ : X Ñ r 0 , 1 s is defined as η ˚ p x q : “ P r Y “ [Equation] | X “ x s “ 131 E r Y | X “ x s ; that is, given an instance X i , the label Y
i is Bernoulli-distributed with mean η p
X i q .
Aswe show in the next section, an optimal classifier can always be written in terms of the true regressionfunction η , motivating estimates p η : X Ñ r 0 , 1 s of η .
Such estimates p η are referred to as “regressors”.
Optimal Classification Beyond Accuracy
A famous result states that classification accuracy is maximized by the “Bayes” classifier
[Equation]
Here, p Y p x q is simply a constant (deterministic) random variable that takes either the value 0 or thevalue 1 with probability 1 (depending on η ˚ p x q ).
Our reason for writing Eq.
(1) in this seeminglyredundant way will become clear with Definition 2 below.
Although η ˚ is unknown in practice, this result is a cornerstone of the statistical theory of binaryclassification because it provides an optimal performance benchmark against which a classifier can beevaluated in terms of accuracy [Devroye et al., 1996, Mitchell, 1997, James et al., 2013].
As discussedpreviously, accuracy can be a poor measure of performance in the imbalanced case.
Therefore, themain contribution of this section, provided in Theorem 3 below, is to generalize this result to abroad class of classification performance measures, including those commonly used in imbalancedclassification.
First, we specify performance measures for which our results apply.
4.1 Confusion Matrix Measures (CMMs)
Nearly all measures of classification performance, including accuracy, precision, recall, F β scores, andothers, can be computed from the confusion matrix, which counts the number of test samples in eachp true class, estimated class q pair.
Formally, let C : “ t C P r 0 , 1 s 2 ˆ 2 : C 1 , 1 ` C 1 , 2 ` C 2 , 1 ` C 2 , 2 “ 1 udenote the set of all possible binary confusion matrices.
Given a classifier p Y , the confusion matrix
C p Y P C and empirical confusion matrix
p C p Y P C are given by
[Equation]
wherein the true positive probability TP p Y and the empirical true positive probability x TP p Y are givenby
[Equation]
and the true and empirical false positive (FP p Y and x FP p Y ), false negative (FN p Y and x FN p Y ), and truepositive (TP p Y and x TP p Y ) probabilities are defined similarly.
Note that the expectation in Eq.
(3) isover randomness both in the data and in the classifier.
Intuitively, measures of a classifier’s performance should improve as TN and TP increase and FN and
FP decrease.
We therefore define the class of Confusion Matrix Measures (CMMs) as follows:
Definition 1 (Confusion Matrix Measure (CMM)) .
A function M : C Ñ R is called a confusionmatrix measure (CMM) if, for any confusion matrix
[Equation]
FN
[Equation]
, we have M p C
[Equation]
Essentially, correcting an incorrect classification should not reduce a CMM.
This is true of anyreasonable measure of classification performance, and hence analyzing CMMs allows us to obtaintheoretical guarantees for all performance measures used in practice.
Specifically, by evaluatingtheir gradients in the directions
[Equation]
and
[Equation]
‰ , one can verify that most performance measures,such as weighted accuracy, precision, recall, F β scores, and Matthew’s Correlation Coefficient are
CMMs.
We note that the area under receiver operating characteristic (AUROC) and area underprecision-recall curve (AUPRC) are not CMMs because they evaluate ( R -valued) scoring functionsrather than ( t 0 , 1 u -valued) classification functions.
However, both AUROC and AUPR are averagesof CMMs computed at various classification thresholds, and, as we discuss in Appendix A.1, ourresults for CMMs thus imply similar results for these measures.
We next present our main result of
Section 4, which generalizes the Bayes classifier (1) to arbitrary CMMs.
4.2 Generalizing
the Bayes Classifier
The Bayes classifier thresholds the regression function deterministically at the value 0 .
Thefollowing generalizes this to a stochastic threshold:
Definition 2 (Regression-Thresholding Classifier (RTC)) .
A classifier p Y :
X Ñ B is called aregression-thresholding classifier (RTC) if, for some p, t P r 0 , 1 s and η : X Ñ r 0 , 1 s ,p Y
[Equation]
„ Bernoulli [Equation] p
[Equation]
, for all x P X . In the sequel, we will denote such classifiers p Y p,t,η , and refer to the pair p p, t q as the threshold .
Now we can state the main result of this paper:
Theorem 3.
For any CMM M and stochastic classifier p Y , there is an RTC p Y p,t,η with M p p Y p,t,η q ě
M p p Y q .
In particular, if M is maximized by any stochastic classifier, then M is maximized by a RTC.
As a special case of Theorem 3, the classical Bayes classifier corresponds to M p C q “ TN ` TP,p “ 0 , and t “ 0 . 5 .
However, as discussed in the next paragraph, without stronger assumptions,
Theorem 3 does not hold for deterministic classifiers.
Since RTCs generalize both the RTC structureand optimality properties of the Bayes classifier, we also refer to them as generalized Bayes classifiers .
We note that existence of any maximizer p Y of M p C p
Y q may depend on specific properties, such as(semi)continuity or convexity of M , which we do not investigate here.
We emphasize that Theorem 3 makes absolutely no assumptions on the distribution of the data.
Inparticular, all prior characterizations of optimal classifiers under general performance metrics assumethat the distribution of the class probability η p X q is absolutely continuous [Narasimhan et al., 2014,2015, Koyejo et al., 2014, Yan et al., 2018, Wang et al., 2019a] 1 , and Wang et al.
[2019a] claimthat regularity assumptions on η p
X q such as absolute continuity “seem to be unavoidable”.
Our
Theorem 3 is the first result to omit such assumptions, and we specifically show that this comes at thecost of the optimal classifier possibly being non-deterministic for a single atom of η p X q .
The generality of Theorem 3 necessitates a significantly more complex proof than prior work.
Inparticular, we prove Theorem 3 in Appendix A using a series of variational arguments.
Roughlyspeaking, given a classifier p Y , we construct a perturbation p Y 1 of p Y
such that either M p C p
Y q ă
M p C p Y 1 q or p Y 1 is an RTC and M p C
p Y q ď M p C p Y 1 q .
Since, the classifier p Y might be quite poorlybehaved (e.g., its behavior on sets of P X -measure 0 could be arbitrary), the technical complexity liesin constructing admissible perturbations (i.e., those that are well-defined classifiers).
For this reason,the proof of Theorem 3 involves a series of constructions of increasingly well-behaved classifiers.
Theorem 3 tells us that a generalized Bayes classifier can always be written in terms of the regressionfunction η and two scalar parameters p p, t q depending on the distribution of η p X q and the CMM M .
The next example shows that this characterization cannot be simplified without stronger assumptions:
Example 4.
Suppose X “ t 0 u is a singleton, η p 0 q P p 0 , 1 q , and, for some θ ą 0 , M p C q “p TP q θ TN.
One can check that M is a valid CMM.
Suppose p Y is an RTC.
It is straightforwardto compute that
M p C p
Y q “ p pη p
qq θ p 1 ´ p qp 1 ´ η p 0 qq 1 t t “ η p 0 qu , and
that M p C p
Y q is uniquelymaximized by p “ θθ ` 1 P p 0 , 1 q and t “ η p 0 q P p 0 , 1 q .
This shows that both threshold parameters pand t in an RTC are necessary, in the absence of further assumptions on M or η .
This example alsoillustrates the need for stochasticity to optimize general CMMs.
Specifically, for any deterministicclassifier p Y , either p Y p 0 q “ 0 (so TP “ 0 ) or p Y p 0 q “ 1 (so TN “ 0 ); in either case, M p C p
Y q “ 0 .
4.3 Relative Performance Guarantees in terms of the Generalized Bayes Classifier
Theorem 3 motivates a two-step approach to imbalanced classification in which one first estimatesthe regression function η and then selects a generalized threshold p p, t q that optimizes empiricalperformance M p p C p Y q .
Such an approach has many practical advantages.
For example, as we showin Appendix D, a simple algorithm can exactly optimize the threshold p p, t q over large datasets in
O p n log n q time.
Additionally, one can address covariate shift or retune a classifier trained under one
CMM to perform well under another CMM, simply by re-optimizing p p, t q , which is statistically andcomputationally much easier than retraining a classifier from scratch.
In this section, we focus on anadvantage for theoretical analysis, namely that the error of such a classifier decomposes into errors inselecting p p, t q and errors in estimating η , allowing the derivation of performance guarantees relativea generalized Bayes classifier.
All results in this section are proven in Appendix B.
We first bound the performance difference of thresholding two regressors in terms of their L 8distance.
This will allow us to bound error due to using a regressor p η instead of the true η .
Lemma 5.
For
[Equation]
[Equation]
Intuitively, Lemma 5 bounds the largest difference in the confusion matrices of p Y p,t,η and p Y p,t,η 1by the probability that the threshold t lies between η and η 1 .
As we will show later, under a marginassumption, this can be bounded by the L 8 distance between η and η 1 .
Our next lemma bounds the worst-case error over thresholds p p, t q P r 0 , 1 s of the empirical confusionmatrix.
This allows us to bound error due to using an empirical threshold p p p, p t q instead of thethreshold p p ˚ , t ˚ q that is optimal for the true regression function.
Lemma 6.
Let η : X Ñ r 0 , 1 s be any regression function.
Then, with probability at least 1 ´ δ ,
[Equation]
Lemma 6 follows from Vapnik-Chervonenkis (VC) bounds on the complexity of the set t p Y p,t,η :p, t P r 0 , 1 su of possible RTCs with fixed regression function η .
In fact, Appendix B proves a moregeneral bound on the error between empirical and true confusion matrices uniformly over any family
F of stochastic classifiers in terms of the growth function of F .
Consequently, when F has finite VCdimension, we obtain uniform convergence at the fast rate a log p n { δ q{ n .
As we formalize later, thissuggests that the difficulty in tuning an imbalanced classifier to optimize a CMM M comes not fromdifficulty in estimating the confusion matrix but rather from the sensitivity of commonly used CMMsto the selected threshold.
Because Theorem 3 shows that any CMM can be optimized by a RTC, westate here only the specific result for RTCs.
Before combining Lemmas 5 and 6 to give the main result of this section, we note a margin assumption,which characterizes separation between the two classes:
Definition 7 (Tsybakov Margin Condition) .
Let C, β ě 0 , t P p 0 , 1 q .
A classification problemwith covariate distribution P X and regression function η satisfies a p C, β q -margin condition around tif, for any ϵ ą 0 , P r| η p X q ´ t |
ď ϵ s ď Cϵ β .
The Tsybakov margin condition, introduced by Mammen and Tsybakov [1999] for t “ 0 . 5 , is widelyused to establish convergence rates for classification in terms of accuracy
[Audibert and Tsybakov,2007, Arlot and Bartlett, 2011, Chaudhuri and Dasgupta, 2014].
Together with the margin conditionand a Lipschitz condition on the M , Lemmas 5 and 6 give the following bound on sub-optimality ofan RTC if the threshold is selected by maximizing M over the empirical confusion matrix:
Corollary 8.
Let η : X Ñ r 0 , 1 s be the true regression function and p η : X Ñ r 0 , 1 s be any regressor.
Let
[Equation]
and p
[Equation]
¯ denote the empirical and true optimal thresholds, respectively.
Suppose M is Lipschitz continuouswith constant L M with respect to the uniform ( L 8 ) metric on C .
Finally, suppose P X and η satisfy ap C, β q -margin condition around t ˚ .
Then, with probability ě 1 ´ δ ,
[Equation]
Uniform Error of the k NN Regressor
In the previous section, we bounded relative performance of an RTC in terms of uniform ( L 8 ) loss ofthe regression function estimate.
Here, we bound uniform loss of one such regressor, the widely usedk -nearest neighbor ( k NN) regressor.
Our analyses include a parameter r , introduced in Section 5,that characterizes a novel sub-type of class imbalance, which we call Uniform Class Imbalance.
Thisleads to insights about how the behavior of the k NN classifier depends not only on the degree, butalso on the structure, of class imbalance in a given dataset.
We begin with some notation:
Definition 9 ( k -Nearest Neighbor Regressor) .
Given a point x P X , let σ p x q denote a permutation of t 1 , ..., n u such that X σ i p x q is the i th -nearest neighbor of x among X 1 , ..., X n .
For integersk P r 1 , n s , the k NN regressor p η k : X Ñ r 0 , 1 s is defined as
[Equation]
We now formalize a novel sub-type of class imbalance:
Definition 10 (Uniform Class Imbalance (UCI)) .
Write the regression function as η “ rζ , wherer P p 0 , 1 s and ζ : X Ñ r 0 , 1 s is a regression function with sup x P X ζ p x q “ 1 .
A classificationproblem has Uniform Class Imbalance (UCI) in the number of samples n if r Ñ 0 as n Ñ 8 .
Intuitively, in UCI, the class Y “ 1 is rare regardless of X .
This includes “difficult” classificationproblems where the covariate X provides only partial information about the class Y and examplesfrom the rare class lie deep within the distribution of the common class.
Examples include raredisease diagnosis [Schaefer et al., 2020] or fraud detection [Awoyemi et al., 2017].
In practice, theclassifier’s role is often to flag “high-risk” samples X , those with η p X q relatively high, for follow-upinvestigation.
UCI can be distinguished from “easier” problems in which, for some x P X , η p X q « 1and so, given enough training data, a classifier can confidently assign the label Y “ 1 .
These includewell-separated classes or deterministic problems (e.g., protein structure prediction; Noé et al.
To our knowledge, such notions of class imbalance have not previously been distinguished.
In theparticular case of data drawn from a logistic model, UCI reduces to the notion of class imbalancedescribed in Wang [2020]; however, UCI applies in more general contexts.
5.1 Uniform Risk Bounds
We now present bounds (proven in Appendix C.1) on uniform error } η ´ p η } 8 “ sup x P X |
η p x q´ p η p x q|of the k NN regressor p η k .
First, recall two standard quantities, covering numbers and shatteringcoefficients, by which we measure complexity of the feature space:
Definition 11 (Covering Number) .
Suppose p X , ρ q is a totally bounded metric space.
Then, forany ϵ ą 0 , the ϵ -covering number N p ϵ q of p X , ρ q is the smallest integer such that there exist N p
ϵ qpoints x 1 , ..., x N p ϵ q P X satisfying X Ď
[Equation]
Definition 12 (Shattering Coefficient) .
For integers n ą 0 , the shattering coefficient of balls inp X , ρ q is S p n q “ sup x 1 ,...,
[Equation]
We now state two assumptions data distribution P X,Y :
Assumption 13 (Dense Covariates Assumption) .
For some p ˚ , ϵ ˚ , d ą 0 , the marginal distribution P X of covariates is lower bounded, for any x P X and ϵ P p 0 , ϵ ˚ s , by P X
p B ϵ p x qq ě p ˚ ϵ d .
Assumption 13 ensures that each query point’s nearest neighbors are sufficiently near to be informative.
We also assume that the regression function ζ is smooth:
Assumption 14 (Hölder Continuity) .
For some α P p 0 , 1 s , L : “ sup x ‰ x 1 P X | ζ p x q´ ζ p x 1 q | ρ α p x,x 1 q ă 8 .
We now state our upper bound on uniform error:
Theorem 15.
Under Assumptions 13 and 14, whenever k { n ď p ˚ p ϵ ˚ q d { 2 , for any δ ą 0 , withprobability at least 1 ´ N ´
[Equation]
{p
[Equation]
˚ n qq 1 { d ¯ e ´ k { 4 ´ δ , we have
[Equation]
If r P O pp log S p n q{ n q , this bound is minimized by k — n , giving } η ´ p η } 8 P O P pp log S
p n qq{ n q .
Otherwise, this bound is minimized by k [Equation] n 2 α[-
-]
[Equation]
` d p log S p n qq [Equation] 2 α
[Equation]
d
[Equation]
d
[Equation]
d , giving
[Equation]
Of the three terms in (5), the first term, of order r p k { n q α { d , comes from smoothing bias of the k NNclassifier.
The second and third terms are due to label noise, with the second term dominating underextreme class imbalance r P
O p log S p n q{ n q and the third term dominating otherwise.
Theorem 5shows that, under UCI, one should use a much larger choice of the tuning parameter k than in the case of balanced classes; indeed, setting k — n
α 2 α ` d p log S
p
n qq d 2 α ` d , which is optimal in the balancedcase, gives a rate that is suboptimal by a factor of r ´ d {p 4 α ` d q .
The following example demonstrates how to apply Theorem 15 in a concrete setting of interest:
Corollary 16 (Euclidean, Absolutely Continuous Case) .
Suppose p X , ρ q “ pr 0 , 1 s d , }¨} 2 q is theunit cube in R d , equipped with the Euclidean metric, and P X has a density that is lower bounded awayfrom 0 on X .
Then, for k — n 2 α 2 α ` d p log n q d 2 α ` d r ´ d 2 α ` d , } η ´ p η } 8 P O P
´ pp log n q{ n q α 2 α ` d r α ` d 2 α ` d ¯ .
The most problematic term in this bound is the exponential dependence on the dimension d of the covariates.
Fortunately, since Theorem 15 utilizes covering numbers, it improves if the covariates exhibitstructure, such as that of a low-dimensional manifold.
We illustrate this in detail in Appendix C.1.
We close with a minimax lower bound, proven in Appendix C.2, on the uniform error of any estimator,over p α, L q -Hölder regression functions.
Up to a polylogarithmic factor in r , the rate of this lowerbound matches that in Theorem 15, suggesting that both bounds are quite tight.
Theorem 17.
Suppose X “ r 0 , 1 s d is the d -dimensional unit cube and X is uniformly distributedon X .
Let Σ
α p
L q denote the family of p α, L q -Hölder continuous regression function.
Then, thereexist constants n 0 and c ą 0 (depending only on α , L , and d ) such that, for all n ě n 0 and anyestimator p η ,
[Equation]
Discussion The upper bounds on } η ´ p η } 8 given in this section can be plugged directly into Corollary 8 to provide error bound under arbitrary CMMs, in terms of the sample size n , hyperparameterk , degree r of UCI, and complexity parameters (margin β , smoothness α , intrinsic dimension d ,etc.) of X and P X,Y .
Thus, these results collectively give some of the first complete finite-sampleguarantees under general performance metrics used for imbalanced classification.
Our analysis showsthat, under severe UCI, the optimal choice of k is much larger than in balanced classification, whereaswe illustrate that this same k leads to sub-optimal, or even inconsistent, estimates of the regressionfunction under other (nonuniform) forms of class imbalance.
Numerical Experiments
We provide two numerical experiments to illustrate our results from Sections 4 and 5.
We repeat eachexperiment 100 times and present average results with 95% confidence intervals computed using thecentral limit theorem.
Python implementations are included as supplementary material.
Experiment 1 Example 4 showed that, under general CMMs, deterministic RTCs are sometimesunable to approach optimal classification performance, necessitating stochastic RTCs.
This experiment demonstrates this gap numerically.
Suppose X “ r 0 , 1 s , over which X is uniformly distributed,and for all x P X , η p x q “ 0 .
5 ¨ 1 t 1 { 3 ď x ă 2 { 3 u ` 1 t 2 { 3 ď x u .η
p x q “ #if x ă 1 {0 .if 1 { 3 ď x ă 2 { if 2 { 3 ď x .
Consider the CMM M p C q “ TP ¨ TN.
Similar to the analysis in Example 4, the optimal valueof M p C q “ p 5 { 12 q 2 “ 25144 is achievable only by a stochastic classifier, whereas as deterministicclassifiers achieve at most M p C q “ 1 { 6 ă 25 { 144 .
For 10 logarithmically spaced values of n between 10 2 and 10 4 , we drew n independent samples ofp X, Y q according the above distribution.
Using this training data, we selected optimal deterministicand stochastic thresholds t P r 0 , 1 s and p t, p q P r 0 , 1 s 2 for the k NN classifier by maximizing
M p p C q over 10 4 uniformly spaced values in r 0 , 1 s and r 0 , 1 s 2 , respectively.
Since, in this example,α “ d “ 1 , we set k “ t n 2 { 3 u as suggested by Theorem 16.
As another point of comparison, we alsoinclude a very different deterministic classifier, a random forest, trained with default parameters of
Python’s scikit-learn package.
We estimated M p C q using 1000 more independently generated
[Figure]
test samples of p X, Y q .
Figure 1a shows regret, i.e., sub-optimality of each classifier relative tothe optimal classifier, in terms of M p C q .
Consistent with our analysis, regrets of the deterministicclassifiers are bounded away from 0 , while regret of the stochastic classifier vanishes as n increases.
Experiment
This experiment demonstrates that making classifiers robust to severe class imbalancerequires distinguishing different sub-types of class imbalance, such as UCI.
Suppose X “ r 0 , 1 s ,
X „ Uniform pr 0 , 1 sq , and r P p 0 , 1 q .
Consider two regression functions η 1 p x q “ r p 1 ´ x q andη 2 p x q “ max t 0 , 1 ´ x { r u .
η 1 and η 2 exhibit the same overall class imbalance, with r { 2 proportionof samples from class 1 .
The regression function η 1 satisfies UCI of degree r , whereas η 2 does notsatisfy a nontrivial degree of UCI.
For sufficiently small r P p 0 , 1 q , specifically r P o ` n ´ d {p 2 α ` 2 d q ˘ ,
Theorem 15 gives that the optimal choice of k under η 1 satisfies k P ω p rn q .
On the other hand, ifk P ω p rn q , then, under η 2 , E r p η k p 0 qs Ñ 0 , so that p η k p 0 q is an inconsistent estimate of η 2 p 0 q “ 1 .
For 10 logarithmically spaced values of n between 10 2 and 10 4 , we drew n independent samples ofp X, Y q according the joint distributions corresponding to each of η 1 and η 2 .
Since, in this example,α “ d “ 1 , to ensure r P o ` n ´ d {p 2 α ` 2 d q ˘ , we set r “ n ´ 1 { 2 .
As indicated by Corollary 16, we setk “ t n 2 { 3 r ´ 1 { 3 u .
We then computed L 8 and L 1 distances between the k NN regressor (Eq.
(4)) andtrue regression function.
We also drew 1000 independent test samples of p X, Y q and used these toestimate the F 1 score of thresholding the k NN regressor at a threshold t determined by optimizing theempirical F 1 score (over the training data) over 100 uniformly-spaced values of t P r 0 , 1 s .
Figure 1bshows the uniform ( L 8 ) error, the average ( L 1 ) error, and the F 1 regret, which we bounded in
Corollary 8.
Consistent with our analysis above, the uniform ( L 8 ) error decays to 0 under η 1 butnot under η 2 .
Meanwhile, the average error ( L 1 ) decays to 0 under both η 1 and η 2 .
Consistent with
Corollary 8, the F 1 regret of the thresholded classifier, which decays to 0 under η 1 but not under η 2 ,mirrors performance of the regressor in uniform ( L 8 ) error rather than average ( L 1 ) error.
Conclusions
Our main conclusions are as follows.
First, without any assumptions on the data-generating distribution, the Bayes-optimal classifier generalizes from accuracy to other performance metrics using astochastic thresholding procedure, while, in general, deterministic classifiers may not achieve Bayesoptimality.
This generalized Bayes classifier provides an optimal performance benchmark relative towhich one can analyze classifiers that threshold estimates of the regression function.
This includes thek NN classifier, for which we provided new guarantees, including minimax-optimally under uniformloss in the presence of Uniform Class Imbalance.
Our results imply that the parameter k needs tobe tuned differently for different sub-types of imbalanced classification, suggesting that developingreliable classifiers for severely imbalanced classes may require a more nuanced understanding ofthe data at hand.
Further work is needed to (a) understand how sub-types of class imbalance can bedistinguished in practice, (b) develop adaptive classifiers that perform well under multiple imbalancesub-types, and (c) extend our results to the multiclass case.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendices A, B,and C.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] Code andinstructions for reproducing the experimental results are included in the supplemental
material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix E.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Derivation of the Generalized Bayes Classifier
In this Appendix, we prove Theorem 3, in which we characterize a stochastic generalization of the
Bayes classifier to arbitrary CMMs.
We reiterate the theorem for the reader here:
Theorem 3.
If argmax p Y P SC M p C p Y q ‰ ∅ , then there exists a regression-thresholding classifier
[Equation]
As described in the main paper, unlike prior results [Koyejo et al., 2014, Yan et al., 2018, Wanget al., 2019b], we do not assume that the distribution of η p X q is absolutely continuous.
This makesproving Theorem considerably more complicated than these previous results.
We prove Theorem 3in a sequence of steps, constructing optimal classifiers in forms progressively closer to that of thegeneralized Bayes classifier described in Theorem 3.
Specifically, we first show, in Lemma 18, thatthere exists an optimal classifier that is a (stochastic) function of the true regression function η ˚ .
We then construct an optimal classifier in which this function of η ˚ is non-decreasing.
Finally, weconstruct an optimal classifier in which this function of η ˚ is a threshold function, as in Theorem 3.
Lemma 18.
For any stochastic classifier p Y : X Ñ B , there is a stochastic classifier p Y 1 : X Ñ B ofthe form
[Equation]
(6) for some f : r 0 , 1 s Ñ r 0 , 1 s , such that C p Y 1 “ C p Y .
Proof.
We start by defining the regression function and discussing formal probability notation.
Let
F “ σ p η ˚ p Z qq be the σ -field generated by the true regression function η ˚ p Z q .
Define p Y 1 : X Ñ Bby
[Equation]
x qq ˙ , where by the η ˚ p x q we are explicitly indicating that η ˚ p x q is the input to the conditional expectation,as the conditional expectation is a measurable function of η ˚ p Z q .
In the sequel, following standardconventions, we omit such notation.
Note that p Y 1 has the desired form and is defined almost surely.
Now, we proceed to get our final notes in place before proceeding with calculations.
Let G “ σ p X qbe the σ -field generated by X .
A key step in the proof is showing the equality
[Equation]
We first provide an intuitive summary.
Observe that conditioned on η ˚ p X q , Y is independent of p Yand p Y 1 separately.
Thus, we can integrate over Y and get the η ˚ p X q .
By the definition of p Y 1 weobtain the conditional expectation with respect to Y , and we have the resulting equality.
For the time being, taking Equation (7) as fact, we complete the rest of the proof.
Combining
Equation (7) and the tower property of conditional expectation, we have
[Equation]
Similarly, one can check that C p Y 1 “ C p Y .
Proof of Equation (7) .
We now present the formal details to establish Equation (7), which maybe skipped if one is uninterested in the measure-theoretic details.
We first to set up the randomvariables needed to formalize the above intuition.
Let S , T , U , V , and W be random variables.
Letp
[Equation]
X q .
We also require U and V to be uniform on r 0 , 1 s .
We need not specify theprecise distribution W .
Further, let p S, T q , U , V , and W all be independent.
For notation purposes,it is convenient for the underlying probability space p Ω , G , P q to be the product probability space ofsome probability spaces to the extent possible.
In particular, we require the set Ω “ Ś Wi “ S Ω i and aset B in G to have a product structure B “ Ś Wi “ S B i for ease of exposition.
Note that the measure P is not a product measure, as S and T are dependent, although it is a product measure with respect tothe laws of p S, T q , U , V , W .
Now, we define Y , p Y 1 , and p Y in terms of U , V , W , and X .
Define
[Equation]
where here U serves as the noise in Y , V serves as the internal randomization of p Y 1 , and W servesas the possible internal randomization of p Y .
Here, 1 is the indicator taking the value 1 if the event incurly braces occurs and 0 otherwise, and r is some function.
Note that this is still of the desired form.
To prove the first part of Equation (7), i.e., to show that the middle term M “ η ˚ p X q E r p Y p
X q| F s “
S E r p Y | F s is indeed the conditional expectation of the left of Equation (7), we must verify twoconditions: (i) M is F -measurable and (ii) the integrals of Y p Y 1 and M are identical on any event
A in F [Durrett, 2010, page 221].
Condition (i) is immediate, since M is a function of η ˚ p
X q andnothing more.
For condition (ii), we do a bit of computation.
Let A “ A S ˆ A T ˆ
A U ˆ A V
ˆ A W be an eventin F .
Note that because A is F -measurable, the sets A U and A V must be all of Ω U and Ω V or theempty set.
Suppose for the moment that both are non-empty.
Then, we have
[Equation]
Note that the crucial first and last equalities in which we break and reform the integral over the entireevent in terms of its coordinates is possible due to Fubini’s theorem for integrable functions.
Thisproves the desired equality when A U “ Ω U and A V “ Ω V , and from the preceding calculation wecan see that the integrals are both 0 when either A U “ H or A V “ H , and so the desired equalityholds.
This establishes the left equality of Equation (7).
Establishing the right hand side of Equation (7) also takes a bit of calculation.
First, we have alreadyverified the measurability condition (i), and so all that remains is to check the integral equality.
Again,let A be an event in F , and assume that A T , A U , and A W are non-empty.
We have
[Equation]
Note that Fubini’s theorem is again used in the first step.
In the event that any of A T , A U , or A Wis empty, then the above equation is 0 and equality holds.
Now, we have to show that the randomvariable M 1 “ ş A T ş A W r p S, T, W q d P W d P T | S is the conditional expectation of p Y with respect to
F .
Measurability is readily apparent, as M 1 is a function of S and no more.
Now, we check thecondition that p Y and M 1 have the same integral on an event A in F .
Again assume that A U and A Ware non-empty, observing in the calculation to follow that equality holds with the value
if either is empty.
Using Fubini’s theorem and some direct computation, we haveż
A p Y d P “ ż
A r p S, T, W q d P “ ż
A S ż
A T ż
A W r p S, T, W q d P W d P T d P S “ ż
A ˆż
A T ż
A W r p S, T, W q d P W d P T ˙
d P “ ż
A M 1 d P .
This completes the proof that the conditional expectation of p Y is M 1 , and so it proves that theconditional expectation of Y p Y is η ˚ p X q E r p Y | F s .
This is the right equality of Equation (7), thuscompleting the proof.
It follows from Lemma 18 that, if M p C p Y q is maximized by any stochastic classifier, then it ismaximized by a classifier p
Y of the form in Eq.
It remains to show that f in Eq.
(6) can be of theform z
ÞÑ p 1 t z “ t u ` 1 t z ą t u for some threshold p p, t q P r 0 , 1 s 2 .
Before proving this, we give asimplifying lemma showing that the problem of maximizing a CMM can be equivalently framed as aparticular functional optimization problem.
This will allow us to to significantly simplify the notationin the subsequent proofs.
Lemma 19.
Let M be a CMM, and suppose
that M p C p
Y q is maximized (over SC ) by a classifier p Yof the form
[Equation]
for some f ˚ : r 0 , 1 s Ñ r 0 , 1 s .
Let f be a solution to the optimization problem
[Equation]
˚
[Equation]
Then, the classifier
[Equation]
also maximizes M p C p Y 1 q (over SC ).
Proof.
This result follows from the definition (Definition 1) of a CMM.
Specifically, by constructionof p Y 1 ,
[Equation]
and
[Equation]
Moreover, since the proportions of positive and negative true labels are independent of the chosenclassifier (i.e., TP p Y 1 ` FN p Y [Equation] TP p Y ` FN p Y and FP p Y 1 ` TN p Y 1 “ FP p Y ` TN p Y ), we have
[Equation]
TP
[Equation]
Y ` ϵ
[Equation]
ȷ
[Equation]
where ϵ 1 : “ FP p Y ´
FP p Y 1 P r 0 , FP p Y s and ϵ 2 : “ TP p Y 1 ´ TP p Y P r 0 , FN s .
Thus, by the definition(Definition 1) of a CMM, M p C p
Y 1 q ě M p C p
Y q .
Lemma 19 essentially shows that maximizing any CMM M is equivalent to performing NeymanPearson classification, at some particular false positive level α depending on M (through f ˚ ) and onthe distribution of η p X q .
For our purposes, this simplifies the remaining steps in proving Theorem 3by allowing us to ignore the details of the particular CMM M and regression function η and focus oncharacterizing solutions to an optimization problem of the form (8) (see, specifically, (9) below).
To characterize solutions to this optimization problem, we will repeatedly utilize the followingmeasure-theoretic technical lemma:
Lemma 20.
Let µ be a measure on r 0 , 1 s with µ pr 0 , 1 sq ą 0 .
Then, there exists z P R such that, forall ϵ ą 0 , µ pr 0 , 1 s X p z
´ ϵ, z qq ą 0 .
Proof.
We prove the contrapositive.
Suppose that, for every z P r 0 , 1 s , there exists ϵ z ą 0 such thatµ pr 0 , 1 s X p z ´ ϵ z , z qq “ 0 .
The family S : “ tr 0 , 1 s X p z ´ ϵ z , z q : z P R u is an open cover ofr 0 , 1 s .
Since r 0 , 1 s is compact, there exists a finite sub-cover S 1 Ď S of r 0 , 1 s .
Thus, by countablesubaddivity of measures,
[Equation]
We are now ready for the main remaining step in the proof of Theorem 3, namely characterizingsolutions of (a generalization of) the optimization problem (8):
Lemma 21.
Let Z be a r 0 , 1 s -valued random variable, and let c P r 0 , 1 s .
Suppose that theoptimization problem
[Equation]
subject to
[Equation]
has a solution.
Then, there is a solution to (9) that is a generalized threshold function.
Proof.
Suppose that there exists a solution f to (9).
We will construct a generalized thresholdfunction that solves (9) in two main steps.
First, we will construct a monotone solution to (9).
Second,we will show that this monotone solution is equal to a generalized threshold function except perhapson a set of probability 0 with respect to Z .
This generalized threshold function is therefore a solutionto (9).
Construction of Monotone Solution to (9) : Define
[Equation]
where the essential supremum and infimum are taken with respect to the measure P Z of Z , with theconventions
g p z q “ 0 whenever P
Z pr 0 , z sq “ 0 and h p
z q “ 1 whenever P Z pp z, 1 sq “ 0 .
We firstshow that, for all z P r 0 , 1 s , g p z q ď h p z q .
We will then use this to show that g “ f except on a set of
P Z measure 0 (i.e., P Z pt z P r 0 , 1 s : g p z q ‰ f p z quq “ 0 ).
Therefore, both E r Zg p Z qs “ E r Zf p
Z qsand E
rp 1 ´
Z q g p Z qs “ E rp 1 ´
Z q f p Z qs .
Since g : r 0 , 1 s Ñ r 0 , 1 s is clearly monotone non-decreasing,the result follows.
Suppose, for sake of contradiction, that, for some z P r 0 , 1 s , g p
z
q ą h p z q .
Then, there exist
A Ď r 0 , z s and B Ď p z, 1 s such that inf z P
A f p z q ą sup z P B f p z q and P Z p
A q , P Z p B q ą 0 .
2 Definez A : “ E r Z | Z P
A s and z B : “ E r Z | Z P B s , and note that, since A Ď r 0 , z s and B Ď p z, 1 s ,
z A ă z B .
Define,
[Equation]
and define ϕ : r 0 , 1 s Ñ r 0 , 1 s by
[Equation]
otherwise, 2 As a concrete example, consider
[Equation]
noting that, by construction of ϵ , ϕ p
z q P r 0 , 1 s for all z P r 0 , 1 s .
Then, by construction of ϕ ,
[Equation]
while
[Equation]
since the function z ÞÑ z 1 ´ z is strictly increasing.
This contradicts the assumption that f optimizes (9),implying g ď h .
We now show that g “ f except on a set of P Z measure 0 .
First, note that, if g p z q ‰ f p z q , theng p z q “ ess sup f pr 0 , z sq “ ess sup f pr 0 , z qq , and so g is left-continuous at z .
For any δ ą 0 , define
A δ : “ t z P r 0 , 1 s : g p z
q ă f p
z
q
´ δ u
and
B δ : “ t z P r 0 , 1 s : g p z q ą f p
z
q
` δ u .
Since
[Equation]
[Equation]
qu “
[Equation]
* and
[Equation]
[Equation]
by countable subadditivity, it suffices to show that P Z p A δ q “ P Z p B
δ q “ 0 for all δ ą 0 .
Suppose, for sake of contradiction, that P Z p
A δ q ą 0 .
Applying Lemma 20 to the measure E ÞÑ
P Z p
A δ X E q , there exists z P R such that, for any ϵ ą 0 , P Z p
A δ X p
z ´
ϵ, z qq ą 0 .
Since g iscontinuous at z , there exists ϵ ą 0 such that g p z
´ ϵ q ě g p z q ´ δ , so that, for all z P
A δ X p
z ´
ϵ, z q ,f p z
q ą g p z q ` δ .
Then, since P Z p A δ X p z ´
ϵ, z qq ą 0 , we have the contradiction
[Equation]
On the other hand, suppose, for sake of contradiction, that P Z p B δ q ą 0 .
Applying Lemma 20 to themeasure E ÞÑ P
Z p B δ X E q
, there exists z P R such that, for any ϵ ą 0 , P Z p B δ X p
z ´
ϵ, z qq ą 0 .
Since g is continuous at z , there exists ϵ ą 0 such that g p z ´ ϵ q ě g p z q ´ δ .
At the same time,since g is non-decreasing, for t P B δ X p z ´ ϵ, z q , f p t q ă g p t q ´ δ ď g p
z
q ´ δ .
Thus, since
P Z p B δ X p
z ´
ϵ, z qq ą 0 , we have h p z
´ ϵ q ă g p z q ´
δ ă g p z
´ ϵ q , contradicting the previouslyshown fact that g ď h .
To conclude, we have shown that P Z pt z P r 0 , 1 s :
g p z q ‰ f p z quq “ 0 .
Construction of a Generalized Threshold Solution: We now construct a solution to (9) that isequal to a generalized threshold function (i.e., a function that has the form p 1 t z “ t u ` 1 t z ą t u )except on a set of P Z -measure 0 .
To show this, it suffices to construct a function f : r 0 , 1 s Ñ r 0 , 1 ssuch that (a) f is monotone non-decreasing and (b) the set f ´ 1 pp 0 , 1 qq is the union of the singletont t u and a set of P Z -measure 0 .
From the previous step of this proof, we may assume that we have a solution f to (9) that is monotonenon-decreasing.
It suffices therefore to show that A : “ f ´ 1 pp 0 , 1 qq is the union of a singleton and aset of P Z -measure 0 .
Define
[Equation]
and t
[Equation]
Then, for all ϵ ą 0 , P Z p A X r 0 , t
´
ϵ sq “ P Z p
A X r t 1 ` ϵ, 1 sq “ 0 .
Hence, if t 0 “ t 1 , then, since
[Equation]
by countable subadditivity, P Z p
A zt t 0 uq “ 0 , which implies that A “ t t 0 u Y p
A zt t 0 uq is the unionof a singleton and a set of measure 0 .
It suffices therefore to prove that t 0 “ t 1 .
It is easy to see, from the definitions of t 0 and t 1 , thatt 0 ď t 1 .
Suppose, for sake of contradiction, that t 0 ă t 1 .
Then, there exists t P p t 0 , t 1 q , and, bydefinition of t 0 and t 1 , both P Z p
A X r 0 , t qq ą 0 and P Z p
A X p t, 1 sq ą 0 .
For any δ ě 0 , define
B δ : “ t z P r 0 , t q : δ ă f p z q ă 1 ´ δ u and
C δ : “ t z P p t, 1 s : δ ă f p z
q ă 1 ´ δ u , so that P Z p B 0 q ą 0 and P Z p C 0 q ą 0 .
By countable subadditivity, there exists δ ą 0 such that
P Z p B δ q ą 0 and P Z p C
δ q ą 0 .
Define ϵ : “ δ ¨ min t P Z p B δ q , P Z p C δ quu ą 0 .
Define g : r 0 , 1 s Ñ R for all z P r 0 , 1 s byg p
z q “ $& % f p z q ´
[Equation]
P
[Equation]
p
[Equation]
if z P B
[Equation]
f p z q `
[Equation]
q if z P C
[Equation]
f p z q otherwise .
and note that, by definition of ϵ , B δ , and C δ , g : r 0 , 1 s Ñ r 0 , 1 s .
Then,
[Equation]
while
[Equation]
Since B δ Ď r 0 , t q and C δ Ď p t, 1 s , this difference is strictly positive, contradicting the assumptionthat f optimizes (9).
Combining Lemma 21 with Lemma 19 completes the proof of our main result, Theorem 3.
A.1 Extension to AUROC
For any regression function η , the receiver operating characteristic (ROC) function ROC η : r 0 , 1 s Ñr 0 , 1 s is
[Equation]
i.e., ROC p x q is the maximum true positive probability (over all regression-thresholding classifierswith regression function η ) achievable while keeping the false positive probability below x .
The areaunder the ROC curve (AUROC) is then given by
[Equation]
[Equation]
While AUROC is not a CMM (as it depends on the entire family of confusion matrices computed atall possible thresholds p p, t q P r 0 , 1 s 2 ), AUROC is widely used to measure performance of classifiersacross the classification thresholds.
Here, we show that our Theorem 3 extends naturally from CMMsto AUROC.
We begin by noting that, for any x P r 0 , 1 s , the performance measure TP ¨ 1 t FP ď x u is a CMM.
Therefore, letting η ˚ denote the true regression function, by Theorem 3, there exists a thresholdp p, t q P r 0 , 1 s 2 such that the regression-thresholding classifier p Y p,t,η ˚ P argmax p Y P SC TP ¨ 1 t FP ď x u ;i.e., p Y p,t,η ˚ maximizes TP ¨ 1 t FP ď x u over all stochastic classifiers.
By definition of ROC (Eq.
(10)),it follows that, for any x P r 0 , 1 s ,
[Equation]
and, by definition AUROC (Eq.
, it then follows that
[Equation]
To conclude, we have shown that thresholding the true regression function is optimal not only underany CMM but also under AUROC.
A identical argument can be made for other performance measures,such as the area under the precision-recall curve (AUPRC), that aggregate CMMs across multipleclassification thresholds.
B Relative Performance Guarantees in terms of the Generalized BayesClassifier
In this Appendix, we prove Lemmas 5 and 6, as well as their consequence, Corollary 8.
Also, in
Section B.1, we demonstrate, in a few key examples, how to compute the Lipschitz constant used in
Corollary 8.
We begin with the proof of Lemma 5, which, at a given threshold p p, t q , bounds the differencebetween the confusion matrices of the true regression function η and an estimate η 1 of η .
We restatethe result for the reader’s convenience:
Lemma 5.
Let p, t P r 0 , 1 s and let η, η 1 : X Ñ r 0 , 1 s .
Then,
[Equation]
Proof.
For the true negative probability, we have
[Equation]
This type of inequality is standard and follows from the fact that, if t lies between η and η 1 , then thedifference of η and t is necessarily less than η and η 1 .
Repeating this calculation for the true positive,false positive, and false negative probabilities gives (12).
Note that, in the presence of degree r Uniform Class Imbalance (see Section 5), one can obtain atighter error bound r P r| η p X q ´ t | ď } η ´ η 1 } 8 s for the true positive and false negative probabilitiesbecause, for all x P X , P r Y “ 1 | X “ x s ď r .
However, the weaker bound (12) simplifies theexposition.
We now turn to proving Lemma 6, which we use to bound the maximum difference between theempirical and true confusion matrices of a regression-thresholding classifier over thresholds p p, t q .
Specifically, we will use this result to bound the difference in confusion matrices between the optimalthreshold p p ˚ , t ˚ q and the threshold p p p, p t q selected by maximizing the empirical CMM.
We actuallyprove a more general version of Lemma 6, for arbitrary classifiers, based on the following definition:
Definition 22 (Stochastic Growth Function) .
Let F be a family of r 0 , 1 s -valued functions on X .
The stochastic growth function Π F : N Ñ N , defined by
[Equation]
n P N
[Equation]
is the maximum number of distinct classifications of n points x 1 , ..., x n by a stochastic classifier p
Ywith p x ÞÑ E
r
p
Y
p
x
qsq P F and randomness given by z 1 , ..., z n .
Definition 22 generalizes the growth function [Mohri et al., 2018], a classical measure of the complexity of a hypothesis class originally due to Vapnik and Chervonenkis [2015], to non-deterministicclassifiers.
Importantly for our purposes, one can easily bound the stochastic growth function ofregression-thresholding classifiers:
Example 23 (Stochastic Growth Function of Regression-Thresholding Classifiers) .
Suppose
F “ t f : X Ñ r 0 , 1 s| for some p, t P r 0 , 1 s , f p x q “ p ¨ 1 t η p x q “ t u ` 1 t η p x q ą t u for all x P X u , so that t p Y f,η : f P F u is the class of regression-thresholding classifiers.
Any set of pointsp x 1 , z 1 q , ..., p x n , z n q , can be sorted in increasing order by η p x q ’s, breaking ties in decreasing order byz ’s. Having sorted the points in this way, t f p x q ą z u “ 0 for the first j points and t f p x q ą z u “ 1for the remaining n ´ j points, for some j P r n s Y t 0 u .
Thus, Π F p n q “ n ` 1 .
We will now prove the following result, from which, together with Example 23, Lemma 6 followsimmediately:
Lemma 6 (Generalized Version) .
Let F be a family of r 0 , 1 s -valued functions on X .
Then, withprobability at least 1 ´ δ ,
[Equation]
Before proving Lemma 6, we note a standard symmetrization lemma, which allows us to replace theexpectation of x TN p Y p,t,η with its value on an independent, identically distributed “ghost sample”.
Lemma 24 (Symmetrization; Lemma 2 of Bousquet et al.
Let X and X 1 be independentrealizations of a random variable with respect to which F is a family of integrable functions.
Then,for any ϵ ą 0 ,
[Equation]
We now use this lemma to prove Lemma 6.
Proof.
To facilitate analyzing the stochastic aspect of the classifier p
Y f,η , let Z 1 , ..., Z n IID „
Uniform pr 0 , 1 sq , such that p Y f,η p
X
i q “ 1 t Z i ă f p η
pp X
i qqu .
Now suppose that we have a ghost sample p X 1 1 , Y 1 1 , Z 1 1 q , ..., p X 1 n , Y 1 n , Z 1 n q .
Let x TN 1 p Y f,η denote theempirical true negative probability computed on this ghost sample, and let x TN p
i q p Y f,η denote the empiricaltrue negative probability computed on
[Equation]
n , Y n , Z n q (i.e., replacing only the i th sample with its ghost).
By the Symmetrization Lemma,
[Equation]
where the second inequality is a union bound over the Π F p 2 n q distinct classifications of 2 n pointsthat can be assigned by p Y f,η with f P F , and the last inequality is from the fact that x TN p Y f,η andx TN 1 p Y f,η are identically distributed and the algebraic fact that, if a ´ b ą ϵ , then either a ´ c ą ϵ { 2 orb ´
c ą ϵ { 2 .
For any particular f P F , by McDiarmid’s inequality [McDiarmid, 1998],
[Equation]
since, for any i P r n s ,
[Equation]
Plugging Inequality (14) into Inequality (13) gives
[Equation]
Repeating this argument with ´ x TN instead of x TN, as well as with x TP, x FN, x FP and their negatives,and taking a union bound over these 8 cases, gives the desired result.
Finally, we will use these two lemmas, together with the margin and Lipschitz assumptions, to prove
Corollary 8, which bounds the sub-optimality of the trained classifier, relative to
the generalized
Bayes classifier, in terms of the desired CMM.
Corollary 8.
Let η : X Ñ r 0 , 1 s denote the true regression function, and let p η : X Ñ r 0 , 1 s denoteany empirical regressor.
Let
[Equation]
¯ and
[Equation]
denote the empirically selected and true optimal thresholds, respectively.
Suppose that M is Lipschitzcontinuous with constant L M with respect to the uniform ( L 8 ) metric on C .
Finally, suppose that
P X and η satisfies a p C, β q -margin condition around t ˚ .
Then, with probability at least 1 ´ δ ,
[Equation]
c
[Equation]
log
[Equation]
Proof.
First, note that
[Equation]
since, by definition of p p p, p t q ,
[Equation]
this term sits between the second and third lines above.
By the Lipschitz assumption,
[Equation] [Equation]
Corollary 8 follows by applying Lemma 5 and the p C,
β q -margin condition to (15) and applying
Lemma 6 to both terms (16) and (17).
B.1 Lipschitz constants for some common CMMs
Corollary 8 assumed that the CMM M was Lipschitz continuous with respect to the sup -norm onconfusion matrices.
In this section, we show how to compute appropriate Lipschitz constants forseveral simple example CMMs.
We begin with a simple example:
Example 25 (Weighted Accuracy) .
For a fixed w P p 0 , 1 q , the w -weighted accuracy is given by
M p C q “ p 1 ´ w q TP ` w TN.
In this case, M clearly has Lipschitz constant L M “ max t w, 1 ´ w u .
For the remainder of this section (only), we will use P : “ E r Y s to denote the positive probabilityof the true labels and p P : “ 1 n ř n i “ 1 Y i to denote the empirical positive probability of the true labels.
Many CMMs of interest, such as Recall and F β scores, are not Lipschitz continuous over all of C .
Fortunately, inspecting the proof of Corollary 8, it suffices for the CMM M to be Lipschitz continuous on the line segments between three specific pairs of confusion matrices, given in Eqs.
(15), (16), and(17).
Deriving the appropriate Lipschitz constants is a bit more complex, and we demonstrate here how to derive them for the specific CMMs of Recall and F β scores.
Of the six confusion matrices in Eqs.
(15), (16), and (17), four are true confusion matrices, whilethe other two are empirical.
The four true confusion matrices have the same positive probability
TP ` FN “ P , which is a function of the true distribution of labels.
The two empirical confusionmatrices have the positive probability x TP ` x FN “ p P , which is a function of the data.
By amultiplicative Chernoff bound, with probability at least 1 ´ e ´ nP { 8 , p P ě P { 2 .
Thus, with highprobability, it suffices for the CMM M to be Lipschitz continuous over confusion matrices withpositive probability at least P { 2 .
For Recall and F β scores, this gives the following Lipschitzconstants:
Example 26 (Recall) .
Recall is given by M p C q “
[Equation]
P .
Thus, M is Lipschitz continuouswith constant L M “ 2 P over the confusion matrices in Eqs.
(15), (16), and (17).
Example 27 ( F β Score) .
For β P p 0 , 8q , the F β score is given by
[Equation]
Hence,
[Equation]
while, since TP ď P ,
[Equation]
Hence, M is Lipschitz continuous with constant 2 p 1 ` β
[Equation]
max
[Equation]
4 ( over the confusion matrices in Eqs.
(15), (16), and (17).
As Examples 26 and 27 demonstrate, the Lipschitz constants of some CMMs can become largewhen the proportion P is positive samples is small.
In particular, when P P O ˆb log nn ˙ , the— L M b log p n { δ q n term of Corollary 8 fails to vanish as n Ñ 8 .
We believe that some loss ofconvergence rate is inevitable if P Ñ 0 as n Ñ 8 , due to the inherent instability of such metrics, butfurther work is needed to understand if the rates given by Corollary 8 are optimal under these metrics.
See also Dembczy ´ nski et al.
[2017] for detailed discussion of Lipschitz constants of many common
CMMs.
C Bounds on Uniform Error of the Nearest Neighbor Regressor
In this appendix, we prove our upper bound on the uniform risk of the k NN regressor (Theorem 15),as well as the corresponding minimax lower bound (Theorem 17).
C.1 Upper Bounds
Here, we prove Theorem 15, our upper bound on the uniform error of the k -NN regressor, restatedbelow:
Theorem 15.
Under Assumptions 13 and 14, whenever k { n ď p ˚ p ϵ ˚ q d { 2 , for any δ ą 0 , withprobability at least 1 ´ N
[Equation]
, we have the uniform error bound
[Equation]
Proof.
For any x P X , let
[Equation]
q denote the mean of the true regression function over the k nearest neighbors of x .
By the triangleinequality,
[Equation]
wherein } η ´ r η k } 8 captures bias due to smoothing and } r η k ´ p η } 8 captures variance due to labelnoise.
We separately show that, with probability at least 1 ´ N
[Equation]
and that, with probability at least 1 ´ δ ,
[Equation]
Bounding the smoothing bias Fix some r ą 0 to be determined, and let t B r p z 1 q , ..., B r p z N p r q qube a covering of p X , ρ q by N p r q balls of radius r , with centers z 1 , ..., z N p r q P X .
By the lower bound assumption on P X , each P X p B r p z j qq ě p ˚ r d .
Therefore, by a multiplicative
Chernoff bound, with probability at least 1 ´ N p r q e ´ p ˚ nr d { 8 , each B r p z
j q contains at least p ˚ nr d { 2samples.
In particular, if r ě ´ 2 k p ˚ n ¯ 1 { d , then each B k contains at least k samples, and it follows that,for every x P X , ρ p x, X σ k p x q q ď 2 r .
Thus, by Hölder continuity of η ,
[Equation]
r [Equation]
Finally, if kn ď p ˚ 2 p r ˚ q d , then we can let r “ ´ 2 k p ˚ n ¯ 1 { d .
Bounding variance due to label noise Let Σ : “ t σ p x q P r n s k : x P X u denote the set of possiblek -nearest neighbor index sets.
One can check from the definition of the shattering coefficient that| Σ | ď S p n q .
For any σ P r n s k , let Z σ : “ ř kj “ 1 Y σ j and let µ σ : “ E r Z σ s .
Note that the conditional random variables Y σ j | X 1 , ..., X n have conditionally independent Bernoulli distributions with means
E r Y σ j | X 1 , ..., X n s “ η p X σ j q and variances E
[Equation]
η p X σ j qq ď r .
Therefore, by Bernstein’s inequality (Eq.
(2.10) of Boucheron et al.
[2013]), for anyϵ ą 0 ,
[Equation]
Moreover, for any x P X , µ σ p x q “ r η k p x q and Z σ p x q { k “ p η p x q .
Hence, by a union bound over σ inΣ ,
[Equation]
Since the right-hand side is independent of X 1 , ..., X n , the unconditional bound
[Equation]
follows.
Plugging in
[Equation]
c
[Equation]
and simplifying gives the final result.
Recall that there is a small (polylogarithmic in r ) gap between our upper and lower bounds.
Webelieve that the upper bound may be slightly loose, and that this might be tightened by using astronger concentration inequality, such as Bennett’s inequality [Bennett, 1962], instead of Bernstein’sinequality in Inequality (19).
Naively applying Theorem 15 results in very slow convergence rates in high dimensions.
For thisreason, we close this section with a corollary of Theorem 15, illustrating that the convergencerates provided by Theorem 15 improve if the covariates are assumed to lie on an (unknown) lowerdimensional manifold:
Corollary 28 (Implicit Manifold Case) .
Suppose Z is a r 0 , 1 s d -valued random variable with adensity lower bounded away from 0 , and suppose that, for some Lipschitz map T : r 0 , 1 s d Ñ R D ,
X “ T p Z q .
Then, N p ϵ q ď p 2 { ϵ q d , and S p n q ď 2 n D ` 1 ` 2 , and so, by Theorem 15, k —
[Equation]
This shows that, if the D covariates lie implicitly on a d -dimensional manifold, convergence ratesdepend on d , which may be much smaller than D .
C.2 Lower Bounds
In this section, we prove Theorem 17, our lower bound on the minimax uniform error of estimating a
Hölder continuous regression function.
We use a standard approach based on the following versionof Fano’s lemma:
Lemma 29.
(Fano’s Lemma; Simplified Form of Theorem 2.5 of Tsybakov 2009)
Fix a family P ofdistributions over a sample space X and fix a pseudo-metric ρ : P ˆ P Ñ r 0 , 8s over P .
Supposethere exist P 0 P P and a set T Ď P such that
[Equation]
where D KL : P ˆ P Ñ r 0 , 8s denotes Kullback-Leibler divergence.
Then,
[Equation]
where the first inf is taken over all estimators p P .
Proof.
We now proceed to construct an appropriate P 0 P P and T Ď P .
Let g : r´ 1 , 1 s d Ñ r 0 , 1 sdefined by
[Equation]
else denote the standard bump function supported on r´ 1 , 1 s d , scaled to have } g } X , 8 “ 1 .
Since g isinfinitely differentiable and compactly supported, it has a finite α -Hölder semi-norm:
[Equation]
where ℓ is any t β u -order multi-index and g ℓ is the corresponding mixed derivative of g .
Define
M : “
[Equation]
ě 1 , since r ě 1 { n .
For each m P r M s d , define g m : X Ñ r 0 , 1 s by
[Equation]
so that t g m : m P r M s d u is a grid of M d bump functions with disjoint supports.
Let ζ 0 ” 14 denotethe constant- 14 function on X .
Finally, for each m P r M s d , define ζ m : X Ñ r 0 , 1 s by
[Equation]
Note that, for any m P r M s d ,
[Equation]
so that ζ m satisfies the Hölder smoothness condition.
For any particular η , let P η denote the jointdistribution of p X, Y q .
Note that P ζ p x, 1 q “ ζ p x q ě 1 { 4 .
Moreover, one can check that, for allx ě ´ 2 { 3 , ´ log p 1 ` x q ď x 2 ´ x .
Hence, for any x P X ,
[Equation] [Equation]
p
[Equation]
p x q ´ ζ m p x qq 2 ´ ζ p x q ` ζ m p x q ¯ , and, similarly
[Equation]
since P ζ p x
[Equation]
0 q “ 1 ´ ζ p x q ě 1 { 4 ,
[Equation] [Equation]
Adding these two terms gives
D KL
[Equation]
where the second inequality comes from the definition of ζ m (Eq. 21) and the third inequality comesfrom the facts that } g } 22 ď 1 and log log x ď 12 log x for all x ą 1 .
Fano’s lemma therefore impliesthe lower bound
[Equation]
where
[Equation]
D Efficient Computation of the Optimal Generalized Threshold
Although the focus of this paper is on statistical properties of regression-thresholding classifiers, wenote that, given an estimate p η of the regression function, the empirically optimal generalized thresholdp p p, p t q , i.e., that which maximizes M p p C p Y p η, p p, p t q , can be efficiently computed.
In this appendix, wedescribe a simple algorithm for doing so.
The key insight is that, because p p p, p t q is used to thresholdthe observed empirical class probabilities p η p X 1 q , ..., p η p X n q before computing M , M p p C
p
Y p η,p,t q onlyneeds to be computed at the n values of p η actually observed in the data.
We also note that, while, by Corollary 8, one can safely use the original training dataset to computep p p, p t q , one can also safely use a much smaller subset of the data, since the rate of convergence in
Lemma 6 is quite fast in n .
Input: Estimated
regression function p η , training covariate
samples
X 1 , ..., X n , CMM M .
Output
: Estimated optimal generalized
threshold p p p, p t q 1 Sample Z 1 , ..., Z n IID „ Uniform pr 0 , 1 sq 2 e 1 , ..., e n Ð p η p X 1 q , ..., p η p X n q 3 p e 1 , Z 1 q , ..., p e n , Z n q Ð LexicographicSort pp e 1 , Z n q , ..., p e n , Z n qq 4 TP Ð 1 n ř ni “ 1 Y i 5 FP Ð 1 ´ TP 6 TN , FN Ð7 p p p, p t q Ð p 0 , 0 q 8 M max Ð M ˆ„
TN
FP
FN
TP ȷ˙ 9 for i “ 1; i ă“ n ; i ` ` do
TP Ð TP ´ Y
i { n
FP Ð TP ´ p 1 ´ Y i q{ n
TN Ð TN ` p 1 ´ Y i q{ n
FN Ð FN ` Y i { n
M new Ð M ˆ„
TN
FP
FN
TP ȷ˙ if M new ą M max then p p p, p t
q Ð p e i , Z i q
M max Ð M new 18 end 19 return p p p, p t q
For large n , the runtime of Algorithm 1 is dominated by Line 3, which involves lexicographicallysorting n pairs.
This can be done in O p n log n q time using standard comparison-based sortingalgorithms.
Hence, the overall runtime of Algorithm 1 is O p n log n q .
E Further Experimental Details
Experiments were run using the numpy and scikit-learn packages in Python 3.9, on a machinerunning Ubuntu 20.04 with an Intel Core i5-9600 CPU and 64 gigabytes of memory.
Instructions forreproducing Figures 1b and 1a are included with the code in the supplementary material.