Original: P REDICTING C LASSIFICATION A CCURACY W HEN A DDING N EW U NOBSERVED C LASSES
Revised: P REDICTING C LASSIFICATION A CCURACY W HEN A DDING N EW U NOBSERVED C LASSES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Multiclass classiﬁers are often designed and evaluated only on a sample from the classes on which they will eventually be applied.
Revised: Multiclass classiﬁers are often designed and evaluated only on a sample from the classes on which they will eventually be applied.

Original: Hence, their ﬁnal accuracy remains unknown.
Revised: Hence, their ﬁnal accuracy remains unknown.

Original: In this work we study how a classiﬁer’s performance over the initial class sample can be used to extrapolate its expected accuracy on a larger, unobserved set of classes.
Revised: In this work we study how a classiﬁer’s performance over the initial class sample can be used to extrapolate its expected accuracy on a larger, unobserved set of classes.

Original: For this, we deﬁne a measure of separation between correct and incorrect classes that is independent of the number of classes: the reversed ROC (rROC), which is obtained by replacing the roles of classes and data-points in the common ROC.
Revised: For this, we deﬁne a measure of separation between correct and incorrect classes that is independent of the number of classes: the reversed ROC (rROC), which is obtained by replacing the roles of classes and data-points in the common ROC.

Original: We show that the classiﬁcation accuracy is a function of the rROC in multiclass classiﬁers, for which the learned representation of data from the initial class sample remains unchanged when new classes are added.
Revised: We show that the classiﬁcation accuracy is a function of the rROC in multiclass classiﬁers, for which the learned representation of data from the initial class sample remains unchanged when new classes are added.

Original: Using these results we formulate a robust neural-network-based algorithm,
Revised: Using these results we formulate a robust neural-network-based algorithm,

Original: CleaneX , which learns to estimate the accuracy of such classiﬁers on arbitrarily large sets of classes.
Revised: CleaneX , which learns to estimate the accuracy of such classiﬁers on arbitrarily large sets of classes.

Original: Unlike previous methods, our method uses both the observed accuracies of the classiﬁer and densities of classiﬁcation scores, and therefore achieves remarkably better predictions than current state-of-the-art methods on both simulations and real datasets of object detection, face recognition, and brain decoding.
Revised: Unlike previous methods, our method uses both the observed accuracies of the classiﬁer and densities of classiﬁcation scores, and therefore achieves remarkably better predictions than current state-of-the-art methods on both simulations and real datasets of object detection, face recognition, and brain decoding.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Advances in machine learning and representation learning led to automatic systems that can identify an individual class from very large candidate sets.
Revised: Advances in machine learning and representation learning led to automatic systems that can identify an individual class from very large candidate sets.

Original: Examples are abundant in visual object recognition (Russakovsky et al., 2015; Simonyan & Zisserman, 2014a), face identiﬁcation (Liu et al., 2017b), and brain-machine interfaces (Naselaris et al., 2011b; Seeliger et al., 2018).
Revised: Examples are abundant in visual object recognition (Russakovsky et al., 2015; Simonyan & Zisserman, 2014), face identiﬁcation (Liu et al., 2017b), and brain-machine interfaces (Naselaris et al., 2011; Seeliger et al., 2018).

Original: In all of these domains, the possible set of classes is much larger than those observed at training or testing.
Revised: In all of these domains, the possible set of classes is much larger than those observed at training or testing.

Original: Acquiring and curating data is often the most expensive component in developing new recognition systems.
Revised: Acquiring and curating data is often the most expensive component in developing new recognition systems.

Original: A practitioner would prefer knowing early in the modeling process whether the datacollection apparatus and the classiﬁcation algorithm are expected to meet the required accuracy levels.
Revised: A practitioner would prefer knowing early in the modeling process whether the datacollection apparatus and the classiﬁcation algorithm are expected to meet the required accuracy levels.

Original: In large multi-class problems, this pilot data may contain considerably fewer classes than would be found when the system is deployed.
Revised: In large multi-class problems, this pilot data may contain considerably fewer classes than would be found when the system is deployed.

Original: (Consider for example the case in which researchers develop a face recognition system that is planned to be used on 10,000 people, but can only collect 1,000 in the initial development phase).
Revised: (Consider for example the case in which researchers develop a face recognition system that is planned to be used on 10,000 people, but can only collect 1,000 in the initial development phase).

Original: This increase in the number of classes changes the difﬁculty of the classiﬁcation problem and therefore the expected accuracy.
Revised: This increase in the number of classes changes the difﬁculty of the classiﬁcation problem and therefore the expected accuracy.

Original: The magnitude of change varies depending on the classiﬁcation algorithm and the interactions between the classes: usually classiﬁcation accuracy will deteriorate as the number of classes increases, but this deterioration is not uniform across classiﬁers and data-distributions.
Revised: The magnitude of change varies depending on the classiﬁcation algorithm and the interactions between the classes: usually classiﬁcation accuracy will deteriorate as the number of classes increases, but this deterioration varies across classiﬁers and data-distributions.

Original: For pilot experiments to work, theory and algorithms are needed to estimate how accuracy of multi-class classiﬁers is expected to change when the number of classes grow.
Revised: For pilot experiments to work, theory and algorithms are needed to estimate how accuracy of multi-class classiﬁers is expected to change when the number of classes grows.

Original: In this work, we develop a prediction algorithm that observes the classiﬁcation results for a small set of classes, and predicts the accuracy on larger class sets.
Revised: In this work, we develop a prediction algorithm that observes the classiﬁcation results for a small set of classes, and predicts the accuracy on larger class sets.

Original: In large multiclass classiﬁcation tasks, a representation is often learned on a set of k 1 classes, whereas the classiﬁer is eventually used on a new larger class set.
Revised: In large multiclass classiﬁcation tasks, a representation is often learned on a set of k 1 classes, whereas the classiﬁer is eventually used on a new larger class set.

Original: On the larger set, classiﬁcation can be performed by applying simple procedures such as measuring the distances in an embedding space between the new example x ∈ X and labeled examples associated with the classes y i ∈ Y .
Revised: On the larger set, classiﬁcation can be performed by applying simple procedures such as measuring the distances in an embedding space between the new example x ∈ X and labeled examples associated with the classes y i ∈ Y .

Original: Such classiﬁers, where the score assigned to a data point x to belong to a class y is independent of the other classes, are deﬁned as marginal classiﬁers (Zheng et al., 2018).
Revised: Such classiﬁers, where the score assigned to a data point x to belong to a class y is independent of the other classes, are deﬁned as marginal classiﬁers (Zheng et al., 2018).

Original: Their performance on the larger set describes how robust the learned representation is.
Revised: Their performance on the larger set describes how robust the learned representation is.

Original: Examples of classiﬁers that are marginal when used on a larger class set include siamese neural networks (Koch et al., 2015), oneshot learning (Fei-Fei et al., 2006) and approaches that directly optimize the embedding (Schroff et al., 2015).
Revised: Examples of classiﬁers that are marginal when used on a larger class set include siamese neural networks (Koch et al., 2015), oneshot learning (Fei-Fei et al., 2006) and approaches that directly optimize the embedding (Schroff et al., 2015).

Original: Our goal in this work is to estimate how well a given marginal classiﬁer will perform on a large unobserved set of k 2 classes, based on its performance on a smaller set of k 1 classes.
Revised: Our goal in this work is to estimate how well a given marginal classiﬁer will perform on a large unobserved set of k 2 classes, based on its performance on a smaller set of k 1 classes.

Original: Recent works (Zheng & Benjamini, 2016; Zheng et al., 2018) set a probabilistic model for rigorously studying this problem, assuming that the k 1 available classes are sampled from the same distribution as the larger set of k 2 classes.
Revised: Recent works (Zheng & Benjamini, 2016; Zheng et al., 2018) set a probabilistic model for rigorously studying this problem, assuming that the k 1 available classes are sampled from the same distribution as the larger set of k 2 classes.

Original: Following the framework they propose, we assume that the sets of k 1 and k 2 classes on which the classiﬁer is trained and evaluated are sampled independently from an inﬁnite continuous set Y according to Y i ∼ P Y ( y ) , and for each class, r data points are sampled independently from X according to the conditional distribution P X | Y ( x | y ) .
Revised: Following the framework they propose, we assume that the sets of k 1 and k 2 classes on which the classiﬁer is trained and evaluated are sampled independently from an inﬁnite continuous set Y according to Y i ∼ P Y ( y ) , and for each class, r data points are sampled independently from X according to the conditional distribution P X | Y ( x | y ) .

Original: In their work, the authors presented two methods for predicting the expected accuracy, one of them originally due to
Revised: In their work, the authors presented two methods for predicting the expected accuracy, one of them originally due to

Original: Kay et al.
Revised: Kay et al.

Original: We cover these methods in Section 2.
Revised: We cover these methods in Section 2.

Original: As a ﬁrst contribution of this work (Section 3), we provide a theoretical analysis that connects the accuracy of marginal classiﬁers to a variant of the receiver operating characteristic (ROC) curve, which is achieved by reversing the roles of classes and data points in the common ROC.
Revised: As a ﬁrst contribution of this work (Section 3), we provide a theoretical analysis that connects the accuracy of marginal classiﬁers to a variant of the receiver operating characteristic (ROC) curve, which is achieved by reversing the roles of classes and data points in the common ROC.

Original: We show that the reversed ROC (rROC) measures how well a classiﬁer’s learned representation separates the correct from the incorrect classes of a given data point.
Revised: We show that the reversed ROC (rROC) measures how well a classiﬁer’s learned representation separates the correct from the incorrect classes of a given data point.

Original: We then prove that the accuracy of marginal classiﬁers is a function of the rROC, allowing the use of well researched ROC estimation methods (Gonc¸alves et al., 2014; Bhattacharya & Hughes, 2015) to predict the expected accuracy.
Revised: We then prove that the accuracy of marginal classiﬁers is a function of the rROC, allowing the use of well researched ROC estimation methods (Gonc¸alves et al., 2014; Bhattacharya & Hughes, 2015) to predict the expected accuracy.

Original: Furthermore, the reversed area under the curve (rAUC) equals the expected accuracy of a binary classiﬁer, where the expectation is taken over all randomly selected pairs of classes.
Revised: Furthermore, the reversed area under the curve (rAUC) equals the expected accuracy of a binary classiﬁer, where the expectation is taken over all randomly selected pairs of classes.

Original: We use our results regarding the rROC to provide our second contribution (Section 4): CleaneX
Revised: We use our results regarding the rROC to provide our second contribution (Section 4): CleaneX

Original: (Classiﬁcation Expected Accuracy Neural EXtrapolation), a new neural-network-based method for predicting the expected accuracy of a given classiﬁer on an arbitrary large set of classes.
Revised: (Classiﬁcation Expected Accuracy Neural EXtrapolation), a new neural-network-based method for predicting the expected accuracy of a given classiﬁer on an arbitrary large set of classes.

Original: CleaneX differs from previous methods by using both the raw classiﬁcation scores and the observed classiﬁcation accuracies for different class-set sizes to calibrate its predictions.
Revised: CleaneX differs from previous methods by using both the raw classiﬁcation scores and the observed classiﬁcation accuracies for different class-set sizes to calibrate its predictions.

Original: In Section 5 we verify the performance of CleaneX on simulations and real data-sets.
Revised: In Section 5 we verify the performance of CleaneX on simulations and real data-sets.

Original: We ﬁnd it achieves better overall predictions of the expected accuracy, and very few ’large’ errors, compared to its competitors.
Revised: We ﬁnd it achieves better overall predictions of the expected accuracy, and very few ’large’ errors, compared to its competitors.

Original: We discuss the implications, and how the method can be used by practitioners, in Section 6.
Revised: We discuss the implications, and how the method can be used by practitioners, in Section 6.

Original: 1. P RELIMINARIES AND NOTATION
Revised: 1. P RELIMINARIES AND NOTATION

Original: In this work x are data points, y are classes, and when referred to as random variables they are denoted by X, Y respectively.
Revised: In this work x are data points, y are classes, and when referred to as random variables they are denoted by X, Y respectively.

Original: We denote by y ( x ) the correct class of x , and use y ∗ when x is implicitly understood.
Revised: We denote by y ( x ) the correct class of x , and use y ∗ when x is implicitly understood.

Original: Similarly, we denote by y (cid:48)
Revised: Similarly, we denote by y (cid:48)

Original: an incorrect class of x .
Revised: an incorrect class of x .

Original: We assume that for each x and y the classiﬁer h assigns a score S y ( x ) , such that the predicted class of x is arg max y S y
Revised: We assume that for each x and y the classiﬁer h assigns a score S y ( x ) , such that the predicted class of x is arg max y S y

Original: ( x ) .
Revised: ( x ) .

Original: On a given dataset of k classes, { y 1 , . . .
Revised: On a given dataset of k classes, { y 1 , . . .

Original: , y k } , the accuracy of the trained classiﬁer h is the probability that it assigns the highest score to the correct class
Revised: , y k } , the accuracy of the trained classiﬁer h is the probability that it assigns the highest score to the correct class

Original: [Equation]
Revised: [Equation]

Original: where P X is the distribution of the data points x in the sample of classes y 1 , . .
Revised: where P X is the distribution of the data points x in the sample of classes y 1 , . .

Original: y k .
Revised: y k .

Original: Since r data points are sampled from each class, P X assumes a uniform distribution over the classes within the given sample.
Revised: Since r data points are sampled from each class, P X assumes a uniform distribution over the classes within the given sample.

Original: An important quantity for a data point x is the probability of the correct class y ∗ to outscore a randomly chosen incorrect class Y (cid:48) ∼ 
Revised: An important quantity for a data point x is the probability of the correct class y ∗ to outscore a randomly chosen incorrect class Y (cid:48) ∼ P Y (cid:48)

Original: [Equation]
Revised: =

Original:  = y ∗ : C x = P Y (cid:48)
Revised: P Y | Y (cid:54) = y ∗ : C x = P Y (cid:48)

Original: ( S y ∗ ( x ) ≥ S y (cid:48)
Revised: ( S y ∗ ( x ) ≥ S y (cid:48)

Original: ( x )) .
Revised: ( x )) .

Original: This is the cumulative distribution function (CDF) of the incorrect scores, evaluated at the value of the correct score.
Revised: This is the cumulative distribution function (CDF) of the incorrect scores, evaluated at the value of the correct score.

Original: We denote the expected accuracy over all possible subsets of k classes from Y by E k
Revised: We denote the expected accuracy over all possible subsets of k classes from Y by E k

Original: [ A ] and its estimator by ˆ E k
Revised: [ A ] and its estimator by ˆ E k

Original: [ A ] .
Revised: [ A ] .

Original: We refer to the curve of E k
Revised: We refer to the curve of E k

Original: [ A ] at different values of k ≥ 2 as the accuracy curve .
Revised: [ A ] at different values of k ≥ 2 as the accuracy curve .

Original: Given a sample of K classes, the average accuracy over all subsets of k classes from the sample, is denoted by ¯ A K k .
Revised: Given a sample of K classes, the average accuracy over all subsets of k classes from the sample, is denoted by ¯ A K k .

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Learning theory provides bounds for multiclass classiﬁcation that depend on the number of classes (Shalev-Shwartz & Ben-David, 2014), and the extension to large mutliclass problems is a topic of much interest (Kuznetsov et al., 2014;
Revised: Learning theory provides bounds for multiclass classiﬁcation that depend on the number of classes (Shalev-Shwartz & Ben-David, 2014), and the extension to large mutliclass problems is a topic of much interest (Kuznetsov et al., 2014;

Original: Lei et al., 2015; Li et al., 2018).
Revised: Lei et al., 2015; Li et al., 2018).

Original: However, these bounds cannot be used to estimate the expected accuracy.
Revised: However, these bounds cannot be used to estimate the expected accuracy.

Original: Generalization to out-of-label set accuracy includes the work of Jain & Learned-Miller (2010).
Revised: Generalization to out-of-label set accuracy includes the work of Jain & Learned-Miller (2010).

Original: The generalization of classiﬁers from datasets with few classes to larger class sets include those of Oquab et al.
Revised: The generalization of classiﬁers from datasets with few classes to larger class sets include those of Oquab et al.

Original: (2014) and Grifﬁn et al.
Revised: (2014) and Grifﬁn et al.

Original: (2007), and are closely related to transfer learning (Pan et al., 2010) and extreme classiﬁcation (Liu et al., 2017a).
Revised: (2007), and are closely related to transfer learning (Pan et al., 2010) and extreme classiﬁcation (Liu et al., 2017a).

Original: More speciﬁc works include that of Abramovich & Pensky (2019), which provides lower and upper bounds for the distance between classes that is required in order to achieve a given accuracy.
Revised: More speciﬁc works include that of Abramovich & Pensky (2019), which provides lower and upper bounds for the distance between classes that is required in order to achieve a given accuracy.

Original: Kay et al.
Revised: Kay et al.

Original: (2008), as adapted by Zheng et al.
Revised: (2008), as adapted by Zheng et al.

Original: (2018), propose to estimate the accuracy of a marginal classiﬁer on a given set of k classes by averaging over x the probability that its correct class outscores a single random incorrect class, raised to the power of k − 1 (the number of incorrect classes in the sample), that is
Revised: (2018), propose to estimate the accuracy of a marginal classiﬁer on a given set of k classes by averaging over x the probability that its correct class outscores a single random incorrect class, raised to the power of k − 1 (the number of incorrect classes in the sample), that is

Original: [Equation]
Revised: [Equation]

Original: Therefore, the expected accuracy can be predicted by estimating the values of C x on the available data.
Revised: Therefore, the expected accuracy can be predicted by estimating the values of C x on the available data.

Original: To do so, the authors propose using kernel density estimation (KDE) choosing the bandwidth with pseudo-likelihood cross-validation (Cao et al., 1994).
Revised: To do so, the authors propose using kernel density estimation (KDE) choosing the bandwidth with pseudo-likelihood cross-validation (Cao et al., 1994).

Original: Zheng et al.
Revised: Zheng et al.

Original: (2018) deﬁne a discriminability function
Revised: (2018) deﬁne a discriminability function

Original: [Equation]
Revised: [Equation]

Original: and show that for marginal classiﬁers, the expected accuracy at k classes is given by
Revised: and show that for marginal classiﬁers, the expected accuracy at k classes is given by

Original: [Equation]
Revised: [Equation]

Original: The authors assume a non-parametric regression model with pre-chosen basis functions b j , so that
Revised: The authors assume a non-parametric regression model with pre-chosen basis functions b j , so that

Original: D ( u ) =
Revised: D ( u ) =

Original: [Equation] j β j b j .
Revised: (cid:80) j β j b j .

Original: To obtain ˆ β the authors minimize the mean squared error (MSE) between the resulting estimation ˆ E k
Revised: To obtain ˆ β the authors minimize the mean squared error (MSE) between the resulting estimation ˆ E k

Original: [ A ] and the observed accuracies ¯ A k 1 k .
Revised: [ A ] and the observed accuracies ¯ A k 1 k .

Original:  R EVERSED ROC
Revised:  R EVERSED ROC

Original: In this section we show that the expected accuracy, E k
Revised: In this section we show that the expected accuracy, E k

Original: [ A ] , can be better understood by studying an ROC-like curve.
Revised: [ A ] , can be better understood by studying an ROC-like curve.

Original: To do so, we ﬁrst recall the deﬁnition of the common ROC: for two classes in a setting in which one class is considered as the positive class and the other as the negative one, the ROC is deﬁned as the graph of the true-positive rate (TPR) against the false-positive rate (FPR) (Fawcett, 2006).
Revised: To do so, we ﬁrst recall the deﬁnition of the common ROC: for two classes in a setting in which one class is considered as the positive class and the other as the negative one, the ROC is deﬁned as the graph of the true-positive rate (TPR) against the false-positive rate (FPR) (Fawcett, 2006).

Original: The common ROC curve represents the separability that a classiﬁer h achieves between data points of the positive class and those of the negative one.
Revised: The common ROC curve represents the separability that a classiﬁer h achieves between data points of the positive class and those of the negative one.

Original: At a working point where the FPR of the classiﬁer is u , we have ROC ( u ) =
Revised: At a working point where the FPR of the classiﬁer is u , we have ROC ( u ) =

Original: TPR (cid:0) [Equation] [Equation] (cid:1) .
Revised: TPR (cid:0) FPR − 1 ( u ) (cid:1) .

Original: In a multiclass setting, we can deﬁne ROC y for each class y by considering y as the positive class, and the union of all other classes as the negative one.
Revised: In a multiclass setting, we can deﬁne ROC y for each class y by considering y as the positive class, and the union of all other classes as the negative one.

Original: An adaptation of the ROC for this setting can be deﬁned as the expectation over the classes of ROC y , that is ROC ( u ) =
Revised: An adaptation of the ROC for this setting can be deﬁned as the expectation over the classes of ROC y , that is ROC ( u ) =

Original: (cid:82) Y ROC y ( u ) dP ( y ) .
Revised: (cid:82) Y ROC y ( u ) dP ( y ) .

Original: In terms of classiﬁcation scores, we have TPR y ( t ) =
Revised: In terms of classiﬁcation scores, we have TPR y ( t ) =

Original: P X ( S y ( x ) > t | y ( x ) =
Revised: P X ( S y ( x ) > t | y ( x ) =

Original: y ) , FPR y ( t ) =
Revised: y ) , FPR y ( t ) =

Original: P X ( S y ( x ) > t | y ( x ) (cid:54) =
Revised: P X ( S y ( x ) > t | y ( x ) (cid:54) =

Original: y ) and thus FPR − 1 y ( u ) =
Revised: y ) and thus FPR − 1 y ( u ) =

Original: sup t { P 
Revised: sup t { P X ( S y ( x ) > t |

Original: [Equation]
Revised: [Equation]

Original: Here, we single out each time one of the classes y and compare the score of the data points that belong to this class with the score of those that do not.
Revised: Here, we single out each time one of the classes y and compare the score of the data points that belong to this class with the score of those that do not.

Original: However, when the number of classes is large, we could instead single out a data point x and compare the score that it gets for the correct class with the scores for the incorrect ones.
Revised: However, when the number of classes is large, we could instead single out a data point x and compare the score that it gets for the correct class with the scores for the incorrect ones.

Original: This reverse view is formalized in the following deﬁnition, where we exchange the roles of data points x and classes y , to obtain the reversed ROC:
Revised: This reverse view is formalized in the following deﬁnition, where we exchange the roles of data points x and classes y , to obtain the reversed ROC:

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Given a data point x , its corresponding reversed true-positive rate is
Revised: Given a data point x , its corresponding reversed true-positive rate is

Original: [Equation]
Revised: [Equation]

Original: The reversed false-positive rate is
Revised: The reversed false-positive rate is

Original: [Equation]
Revised: [Equation]

Original: and accordingly
Revised: and accordingly

Original: [Equation]
Revised: [Equation]

Original: Consequently, the reversed ROC is
Revised: Consequently, the reversed ROC is

Original: [Equation]
Revised: [Equation]

Original: and the average reversed ROC is
Revised: and the average reversed ROC is

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: Note that dP ( x ) in Equation 9 assumes a uniform distribution with respect to a given sample of classes { y 1 , . . .
Revised: Note that dP ( x ) in Equation 9 assumes a uniform distribution with respect to a given sample of classes { y 1 , . . .

Original: , y k } and their corresponding data points.
Revised: , y k } and their corresponding data points.

Original: Since P Y (cid:48) ( S y (cid:48) ( x ) > t ) is a decreasing function of t , it can be seen that rROC x ( u ) = 

Revised: Since P Y (cid:48) ( S y (cid:48) ( x ) > t ) is a decreasing function of t , it can be seen that rROC x ( u ) =

Original: iff u >
Revised: iff u >

Original: P Y (cid:48)
Revised: P Y (cid:48)

Original: ( S y (cid:48)
Revised: ( S y (cid:48)

Original: ( x ) > S y ∗ )
Revised: ( x ) > S y ∗ )

Original: = 1 − C x (cf.
Revised: = 1 − C x (cf.

Original: Proposition 1 in Appendix A).
Revised: Proposition 1 in Appendix A).

Original: However, even though rROC x is a step function, the rROC resembles a common ROC curve, as illustrated in Figure 1. 
Revised: However, even though rROC x is a step function, the rROC resembles a common ROC curve, as illustrated in Figure 1. [Figure] [Equation]

Original: [Figure]
Revised: [Figure]

Original: 3. T HE REVERSED ROC AND CLASSIFICATION ACCURACY
Revised: 3. T HE REVERSED ROC AND CLASSIFICATION ACCURACY

Original: In what follows we show that the classiﬁcation accuracy can be expressed using the average reversed
Revised: In what follows, we show that the classiﬁcation accuracy can be expressed using the average reversed

Original: ROC curve.
Revised: ROC curve.

Original: We assume a marginal classiﬁer which assigns scores without ties, that is for all x and all y i (cid:54) =
Revised: We assume a marginal classiﬁer which assigns scores without ties, that is for all x and all y i (cid:54) =

Original: y j we have S y i ( x ) (cid:54) =
Revised: y j we have S y i ( x ) (cid:54) =

Original: S y j
Revised: S y j

Original: ( x ) almost surely.
Revised: ( x ) almost surely.

Original: In such cases the following theorem holds:
Revised: In such cases the following theorem holds:

Original: Theorem 1.
Revised: Theorem 1.

Original: The expected balanced classiﬁcation accuracy at k classes is
Revised: The expected balanced classiﬁcation accuracy at k classes is

Original: [Equation]
Revised: [Equation]

Original: To prove this theorem we show that
Revised: To prove this theorem we show that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and the rest follows immediately from the results of Zheng et al.
Revised: and the rest follows immediately from the results of Zheng et al.

Original: (see Equation 4).
Revised: (see Equation 4).

Original: We provide the detailed proof in Appendix A.
Revised: We provide the detailed proof in Appendix A.

Original: Now, using the properties of the rROC we get
Revised: Now, using the properties of the rROC we get

Original: [Equation]
Revised: [Equation]

Original: Therefore, in order to predict the expected accuracy it sufﬁces to estimate the values of C x .
Revised: Therefore, in order to predict the expected accuracy it sufﬁces to estimate the values of C x .

Original: A consequence of the result above is that the expressions that Kay et al.
Revised: A consequence of the result above is that the expressions that Kay et al.

Original: (2008) and Zheng et al.
Revised: (2008) and Zheng et al.

Original: (2018) estimate (Equations 2 and 4 respectively) are in fact the same.
Revised: (2018) estimate (Equations 2 and 4 respectively) are in fact the same.

Original: Nevertheless, their estimation methods differ signiﬁcantly.
Revised: Nevertheless, their estimation methods differ signiﬁcantly.

Original: Finally, we note that the theoretical connection between the reversed ROC and the evaluation of classiﬁcation models extends beyond this particular work.
Revised: Finally, we note that the theoretical connection between the reversed ROC and the evaluation of classiﬁcation models extends beyond this particular work.

Original: For example, by plugging k = 2 into
Revised: For example, by plugging k = 2 into

Original: Theorem 1 it immediately follows that the area under the reversed ROC curve (rAUC) is the expected balanced accuracy of two classes: rAUC := {+(cid:82) 1 
 rROC ( u ) du =+}
Revised: Theorem 1 it immediately follows that the area under the reversed ROC curve (rAUC) is the expected balanced accuracy of two classes: rAUC := {+(cid:82) 1 
 rROC ( u ) du =+}

Original: [Equation]{+E 
 [ A ] .+}
Revised: {+E 
 [ A ] .+}

Original:  E XPECTED ACCURACY PREDICTION
Revised:  E XPECTED ACCURACY PREDICTION

Original: In this section we present a new algorithm, CleaneX , for the prediction of the expected accuracy of a classiﬁer.
Revised: In this section we present a new algorithm, CleaneX , for the prediction of the expected accuracy of a classiﬁer.

Original: The algorithm is based on a neural network that estimates the values of C x using the classiﬁer’s scores on data from the available k 1 classes.
Revised: The algorithm is based on a neural network that estimates the values of C x using the classiﬁer’s scores on data from the available k 1 classes.

Original: These estimations are then used, based on the results of the previous section, to predict the classiﬁcation accuracy at k 2 > k 1 classes.
Revised: These estimations are then used, based on the results of the previous section, to predict the classiﬁcation accuracy at k 2 > k 1 classes.

Original: The general task of estimating densities using neural networks has been widely addressed (MagdonIsmail & Atiya, 1999;
Revised: The general task of estimating densities using neural networks has been widely addressed (MagdonIsmail & Atiya, 1999;

Original: Papamakarios et al., 2017; Dinh et al., 2016; Uria et al., 2014).
Revised: Papamakarios et al., 2017; Dinh et al., 2016; Uria et al., 2014).

Original: However, in our case, we need to estimate the cumulative distribution only at the value of the correct score
Revised: However, in our case, we need to estimate the cumulative distribution only at the value of the correct score

Original: S y ∗ ( x ) .
Revised: S y ∗ ( x ) .

Original: This is an easier task to perform and it allows us to design an estimation technique that learns to estimate the CDF in a supervised manner.
Revised: This is an easier task to perform and it allows us to design an estimation technique that learns to estimate the CDF in a supervised manner.

Original: We use a neural network f ( · ; θ ) : R k 1 → R whose input is a vector of the correct score, and k 1 − 1 incorrect scores, and its output is ˆ C x .
Revised: We use a neural network f ( · ; θ ) : R k 1 → R whose input is a vector of the correct score, and k 1 − 1 incorrect scores, and its output is ˆ C x .

Original: Once ˆ C x values are estimated for each x , an expected accuracy for each 2 ≤ k ≤ k 1 can be estimated according to Equation (12): ˆ E k
Revised: Once ˆ C x values are estimated for each x , an expected accuracy for each 2 ≤ k ≤ k 1 can be estimated according to Equation (12): ˆ E k

Original: [ A ] = 1 N [Equation] − 1 , where N = rk 1 .
Revised: [ A ] = 1 N (cid:80) x ˆ C xk − 1 , where N = rk 1 .

Original: In each iteration, the network’s weights are optimized by minimizing the average over 2 ≤ k ≤ k 1 square distances from the observed accuracies ¯ A k1 k :
Revised: In each iteration, the network’s weights are optimized by minimizing the average over 2 ≤ k ≤ k 1 square distances from the observed accuracies ¯ A k

Original: [Equation]
Revised: [Equation]

Original: [-After the weights have converged, the estimated expected accuracy at k 2 > k 1 classes can be obtained by ˆ E k 
-]
Revised: [-After the weights have converged, the estimated expected accuracy at k 2 > k 1 classes can be obtained by ˆ E k 
-][Equation] (cid:16) [Equation]

Original: [ A ] = 1 N 
Revised: N (cid:80)

Original: [Equation]
Revised: [Equation]

Original: x
Revised: k −

Original: [Equation]
Revised: [Equation]

Revised: − ¯ A k 1 k (cid:17) 2 .

Original: {+After the weights have converged, the estimated expected accuracy at k 2 > k 1 classes can be obtained by ˆ E k 
+}
Revised: {+After the weights have converged, the estimated expected accuracy at k 2 > k 1 classes can be obtained by ˆ E k 
+}

Revised: [ A ] = 1 N (cid:80) x

Revised: ˆ C xk 2 − 1 .

Original: Note that regardless of the target k 2 , the networks input size is k 1 .
Revised: Note that regardless of the target k 2 , the networks input size is k 1 .

Original: The proposed method is described in detail in Algorithm 1.
Revised: The proposed method is described in detail in Algorithm 1.

Original: Unlike KDE and non-parametric regression, our method does not require a choice of basis functions or kernels.
Revised: Unlike KDE and non-parametric regression, our method does not require a choice of basis functions or kernels.

Original: As will be seen in the next sections, in all of our experiments we use the samenetwork architecture and hyperparameters, indicating that our algorithm requires very little tuning when applied to new classiﬁcation problems.
Revised: It does require choosing the network architecture, but as will be seen in the next sections, in all of our experiments we use the same architecture and hyperparameters, indicating that our algorithm requires very little tuning when applied to new classiﬁcation problems.

Original: Although the neural network allows more ﬂexibility compared to the non-parametric regression, the key difference between the method we propose and previous ones is that CleaneX combines two sources of information: the classiﬁcation scores (which are used as the network’s inputs) and the average accuracies of the available classes (which are used in the minimized objective).
Revised: Although the neural network allows more ﬂexibility compared to the non-parametric regression, the key difference between the method we propose and previous ones is that CleaneX combines two sources of information: the classiﬁcation scores (which are used as the network’s inputs) and the average accuracies of the available classes (which are used in the minimized objective).

Original: Unlike the
Revised: Unlike the

Original: KDE and regression based methods which use only one type of information, we estimate the CDF of the incorrect scores evaluated at the correct score, C x , but calibrate the estimations to produce accuracy curves that ﬁt the observed ones.
Revised: KDE and regression based methods which use only one type of information, we estimate the CDF of the incorrect scores evaluated at the correct score, C x , but calibrate the estimations to produce accuracy curves that ﬁt the observed ones.

Original: We do so by exploiting the mechanism of neural networks to iteratively optimize an objective function over the whole dataset.
Revised: We do so by exploiting the mechanism of neural networks to iteratively optimize an objective function over the whole dataset.

Original:  A endfor  do
Revised: Compute ¯ A endfor each x training set do

Original: Set S (cid:48)
Revised: Set S (cid:48)

Original: ( x ) ← (cid:16) S y (cid:48) 1 ( x ) , . . . 
Revised: ( x ) ← (cid:16) S y (cid:48) 1 ( x ) , . . . ,

Original: , S y (cid:48)
Revised: S y (cid:48)

Original: [Equation] ( x ) (cid:17) Sort S (cid:48)
Revised: k 1 − 1 ( x ) (cid:17) Sort S (cid:48)

Original: ( x )
Revised: ( x )

Original: Set S x ← ( S y ∗ ( x ) , S (cid:48)  x )) endfor j = 1 , . . 
Revised: Set S x ← ( S y ∗ ( x ) , S (cid:48) ( x )) endfor j = 1 , . . .

Original: , J do  do
Revised: , J do for x do

Original: Set ˆ C x ←f ( S x ; θ j ) end
Revised: Set ˆ C x ← end

Original: [Equation]
Revised: [Equation]

Original: (cid:17)
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: end
Revised: end

Original: Return ˆ Ek 2 [ A] = 1 N (cid:80) x ˆ C xk
Revised: Return ˆ E 2 [ A = 1 N (cid:80) x ˆ C xk

Original: [Equation]

Original: − 1 ;
Revised: − 1 ;

Original:  S IMULATIONS AND EXPERIMENTS
Revised:  S IMULATIONS AND EXPERIMENTS

Original: Here we compare CleaneX to the KDE method (Kay et al., 2008) and to the non-parametric regression (Zheng et al., 2018) on simulated datasets that allow to examine under which conditions each method performs better, and on real-world datasets.
Revised: Here we compare CleaneX to the KDE method (Kay et al., 2008) and to the non-parametric regression (Zheng et al., 2018) on simulated datasets that allow to examine under which conditions each method performs better, and on real-world datasets.

Original: Our goal is to predict the expected accuracy
Revised: Our goal is to predict the expected accuracy

Original: E k
Revised: E k

Original: [ A ] ∈ [0 , 1] for different values of k .
Revised: [ A ] ∈ [0 , 1] for different values of k .

Original: Therefore, for each method we compute the estimated expected accuracies ˆ E k
Revised: Therefore, for each method we compute the estimated expected accuracies ˆ E k

Original: [ A ] for 2 ≤ k ≤ k 2 and measure the success of the prediction using the root of the mean integrated square error (RMSE): (cid:0) 
Revised: [ A ] for 2 ≤ k ≤ k 2 and measure the success of the prediction using the root of the mean integrated square error (RMSE): (cid:0) 1 k 2 − 1 (cid:80) k 2 k

Original: [Equation]
Revised: [Equation]

Original: For our method, we use in all the experiments an identical feed-forward neural network with two hidden layers of sizes 512 and 128, rectiﬁed linear activation between the layers, and sigmoid activation applied on the output.
Revised: For our method, we use in all the experiments an identical feed-forward neural network with two hidden layers of sizes 512 and 128, rectiﬁed linear activation between the layers, and sigmoid activation applied on the output.

Original: We train the network according to Algorithm 1 for J = 10 ,iterations with learning rate of η = 10 − 4 using Adam optimizer (Kingma & Ba, 2014).
Revised: We train the network according to Algorithm 1 for J = 10 ,iterations with learning rate of η = 10 − 4 using Adam optimizer (Kingma & Ba, 2014).

Original: For the regression based method we choose a radial basis and for the KDE based method a normal kernel, as recommended by the authors.
Revised: For the regression based method we choose a radial basis and for the KDE based method a normal kernel, as recommended by the authors.

Original: The technical implementation details are provided in Appendix C.
Revised: The technical implementation details are provided in Appendix C.

Original: A naive calculation of ¯
Revised: A naive calculation of ¯

Original: A Kk requires averaging the obtained accuracy over all possible subsets of k classes from K classes.
Revised: A Kk requires averaging the obtained accuracy over all possible subsets of k classes from K classes.

Original: However, Zheng et al.
Revised: However, Zheng et al.

Original: (2018) showed that for marginal classiﬁers ¯ A Kk = [Equation]
Revised: (2018) showed that for marginal classiﬁers ¯ A Kk =( Kk ) 1 rk (cid:80) x (cid:0)

Original:  , where the sum is taken over all data points x from the K classes, and R x is the number of incorrect classes that the correct class of x outperforms: 
Revised: R x k − 1 (cid:1) , where the sum is taken over all data points x from the K classes, and R x is the number of incorrect classes that the correct class of x outperforms: R x = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: x
Revised: > S

Original: [Equation]
Revised: [Equation]

Revised: ( x ) } .

Original: We use this result to compute ¯ A k 2 k values.
Revised: We use this result to compute ¯ A k 2 k values.

Original: 5. S IMULATION S TUDIES
Revised: 5. S IMULATION S TUDIES

Original: Here we provide comparison of the methods under different parametric settings.
Revised: Here we provide comparison of the methods under different parametric settings.

Original: We simulate both classes and data points as d -dimensional vectors, with d = 5 (and d = 3 , 10 shown in Appendix
Revised: We simulate both classes and data points as d -dimensional vectors, with d = 5 (and d = 3 , 10 shown in Appendix

Original: B).
Revised: B).

Original: Settings vary in the distribution of classes Y and data-points X | Y , and in the spread of datapoints around the class centroids.
Revised: Settings vary in the distribution of classes Y and data-points X | Y , and in the spread of datapoints around the class centroids.

Original: We sample the classes y 1 , . . .
Revised: We sample the classes y 1 , . . .

Original: , y k 2 from a multivariate normal distribution N (0 , I ) or a multivariate uniform distribution U ( −√ 3 , √ 3) 1 .
Revised: , y k 2 from a multivariate normal distribution N (0 , I ) or a multivariate uniform distribution U ( −√ 3 , √ 3) 1 .

Original: We sample r = 10 data points for each class, either from a multivariate normal distribution N ( y, σ 2 I ) or from a multivariate uniform distribution U ( y − √ 3 σ 2 , y + √ 3 σ 2 ) .
Revised: We sample r = 10 data points for each class, either from a multivariate normal distribution N ( y, σ 2 I ) or from a multivariate uniform distribution U ( y − √ 3 σ 2 , y + √ 3 σ 2 ) .

Original: The difﬁculty level is determined by σ 2 = 0 . 1 , 0 . 2 .
Revised: The difﬁculty level is determined by σ 2 = 0 . 1 , 0 . 2 .

Original: The classiﬁcation scores S y
Revised: The classiﬁcation scores S y

Original: ( x ) are set to be the euclidean distances between x and y (in this case the correct scores are expected to be lower than the incorrect ones, entailing some straightforward modiﬁcations to our analysis).
Revised: ( x ) are set to be the euclidean distances between x and y (in this case the correct scores are expected to be lower than the incorrect ones, entailing some straightforward modiﬁcations to our analysis).

Original: For each classiﬁcation problem, we subsample 50 times k 1 classes, for k 1 = 100 , 500 , and predict the accuracy at 2 ≤ k ≤ k 2 = 2000 classes.
Revised: For each classiﬁcation problem, we subsample 50 times k 1 classes, for k 1 = 100 , 500 , and predict the accuracy at 2 ≤ k ≤ k 2 = 2000 classes.

Original: [Figure]
Revised: [Figure]

Original: than 0.05 and are therefore not shown.
Revised: 0.05 and are therefore not shown.

Original: The results of the simulations, summarized in Figure 2, show the distribution of RMSE values for each method and setting over 50 repetitions.
Revised: The results of the simulations, summarized in Figure 2, show the distribution of RMSE values for each method and setting over 50 repetitions.

Original: The corresponding accuracy curves are shown in Figure 4 in Appendix B. It can be seen that extrapolation to 20 times the original number of classes can be achieved with reasonable errors (the median RMSE is less than 5% in almost all scenarios).
Revised: The corresponding accuracy curves are shown in Figure 4 in Appendix B. It can be seen that extrapolation to 20 times the original number of classes can be achieved with reasonable errors (the median RMSE is less than 5% in almost all scenarios).

Original: Our method performs better or similar to the competing methods, often with substantial gain.
Revised: Our method performs better or similar to the competing methods, often with substantial gain.

Original: For example, for d = 5 the results show that our method outperforms the KDE based method in all cases; it outperforms the regression based method in 7 out of 8 settings for k 1 = 100 , and for k 1 = 500 the regression based method and CleaneX achieve very similar results.
Revised: For example, for d = 5 the results show that our method outperforms the KDE based method in all cases; it outperforms the regression based method in 7 out of 8 settings for k 1 = 100 , and for k 1 = 500 the regression based method and CleaneX achieve very similar results.

Original: As can be seen in Figures 2 and 4, the accuracy curves produced by the KDE method are highly biased and therefore it consistently achieves the worst performance.
Revised: As can be seen in Figures 2 and 4, the accuracy curves produced by the KDE method are highly biased and therefore it consistently achieves the worst performance.

Original: The results of the regression method, on the other hand, are more variable than our method, especially at k 1 = 100 .
Revised: The results of the regression method, on the other hand, are more variable than our method, especially at k 1 = 100 .

Original: Additional results for d = 3 , 10 (see Appendix
Revised: Additional results for d = 3 , 10 (see Appendix

Original: B) are consistent with these results, though all methods predict better for d = 10 .
Revised: B) are consistent with these results, though all methods predict better for d = 10 .

Original: 5. E XPERIMENTS
Revised: 5. E XPERIMENTS

Original: Here we present the results of three experiments performed on datasets from the ﬁelds of computer vision and computational neuroscience.
Revised: Here we present the results of three experiments performed on datasets from the ﬁelds of computer vision and computational neuroscience.

Original: We repeat each experiment 50 times.
Revised: We repeat each experiment 50 times.

Original: In each repetition we sub-sample k 1 classes and predict the accuracy at 2 ≤ k ≤ k 2 classes.
Revised: In each repetition we sub-sample k 1 classes and predict the accuracy at 2 ≤ k ≤ k 2 classes.

Original: Experiment 1 - Object Detection (CIFAR-100)
Revised: Experiment 1 - Object Detection (CIFAR-100)

Original: We use the CIFAR dataset (Krizhevsky et al., 2009) that consists of 32 × 32 color images from 100 classes, each class containing 600 images.
Revised: We use the CIFAR dataset (Krizhevsky et al., 2009) that consists of 32 × 32 color images from 100 classes, each class containing 600 images.

Original: Each image is embedded into 512-dimensional space by a VGG-16 network (Simonyan & Zisserman, 2014b) which was pre-trained on the ImageNet dataset (Deng et al., 2009).
Revised: Each image is embedded into 512-dimensional space by a VGG-16 network (Simonyan & Zisserman, 2014) which was pre-trained on the ImageNet dataset (Deng et al., 2009).

Original: On the training set, the centroid of each class is calculated and the classiﬁcation scores for each image in the test set are set to be the distances of the image embedding from the centroids.
Revised: On the training set, the centroid of each class is calculated and the classiﬁcation scores for each image in the test set are set to be the distances of the image embedding from the centroids.

Original: The classiﬁcation accuracy is extrapolated from k 1 = 10 to k 2 = 100 classes.
Revised: The classiﬁcation accuracy is extrapolated from k 1 = 10 to k 2 = 100 classes.

Original: Experiment 2 - Face Recognition (LFW)
Revised: Experiment 2 - Face Recognition (LFW)

Original: We use the “Labeled Faces in the Wild” dataset (Huang et al., 2007) and follow the procedure described in Zheng et al.
Revised: We use the “Labeled Faces in the Wild” dataset (Huang et al., 2007) and follow the procedure described in Zheng et al.

Original: (2018): we restrict the dataset to the 1672 individuals for whom it contains at least 2 face photos and include in our data exactly 2 randomly chosen photos for each person.
Revised: (2018): we restrict the dataset to the 1672 individuals for whom it contains at least 2 face photos and include in our data exactly 2 randomly chosen photos for each person.

Original: We use one of them as a label y , and the other as a data point x , consistent with a scenario of single-shot learning.
Revised: We use one of them as a label y , and the other as a data point x , consistent with a scenario of single-shot learning.

Original: Each photo is embedded into 128dimensional space using OpenFace embedding (Amos et al., 2016).
Revised: Each photo is embedded into 128dimensional space using OpenFace embedding (Amos et al., 2016).

Original: The classiﬁcation scores are set to be the euclidean distance between the embedding of each photo and the photos that are used as labels.
Revised: The classiﬁcation scores are set to be the euclidean distance between the embedding of each photo and the photos that are used as labels.

Original: Classiﬁcation accuracy is extrapolated from k 1 = 200 to k 2 = 1672 classes.
Revised: Classiﬁcation accuracy is extrapolated from k 1 = 200 to k 2 = 1672 classes.

Original: Experiment 3 - Brain Decoding (fMRI)
Revised: Experiment 3 - Brain Decoding (fMRI)

Original: We analyze a “mind-reading” task described by Kay et al.
Revised: We analyze a “mind-reading” task described by Kay et al.

Original: (2008) in which a vector of v = 1250 neural responses is recorded while a subject watches n =natural images inside a functional MRI.
Revised: (2008) in which a vector of v = 1250 neural responses is recorded while a subject watches n =natural images inside a functional MRI.

Original: The goal is to identify the correct image from the neural response vector.
Revised: The goal is to identify the correct image from the neural response vector.

Original: Similarly to the decoder in (Naselaris et al., 2011a), we use n t = 750 images and their response vectors to ﬁt an embedding g ( · ) of images into the brain response space, and to estimate the residual covariance Σ .
Revised: Similarly to the decoder in (Naselaris et al., 2011), we use n t = 750 images and their response vectors to ﬁt an embedding g ( · ) of images into the brain response space, and to estimate the residual covariance Σ .

Original: The remaining n − n t =
Revised: The remaining n − n t =

Original: k 2 = 1000 examples are used as an evaluation set for g ( · ) .
Revised: k 2 = 1000 examples are used as an evaluation set for g ( · ) .

Original: For image y and brain vector x , the score is the negative Mahalanobis distance −(cid:107) g
Revised: For image y and brain vector x , the score is the negative Mahalanobis distance −(cid:107) g

Original: ( y ) − x (cid:107) 2Σ .
Revised: ( y ) − x (cid:107) 2Σ .

Original: For each response vector, the image with the highest score is selected.
Revised: For each response vector, the image with the highest score is selected.

Original: k 1 = 200 examples from the evaluation set are extrapolated to k 2 = 1000 .
Revised: k 1 = 200 examples from the evaluation set are extrapolated to k 2 = 1000 .

Original: The experimental results, summarized in Figure 3.D, show that generally CleaneX outperforms the regression and KDE methods: Figure 3.E shows that the median ratio between RMSE values of the competing methods and CleaneX is higher than 1 in all cases except for KDE on the LFW dataset.
Revised: The experimental results, summarized in Figure 3.D, show that generally CleaneX outperforms the regression and KDE methods: Figure 3.E shows that the median ratio between RMSE values of the competing methods and CleaneX is higher than 1 in all cases except for KDE on the LFW dataset.

Original: As can be seen in Figure 3.A and B, the estimated accuracy curves produced by CleaneX have lower variance than those of the regression method (which do not always decrease with k ), and on average our method outperforms the regression, achieving lower prediction errors by 18% (CIFAR), 32% (LFW) and 22% (fMRI).
Revised: As can be seen in Figure 3.A and B, the estimated accuracy curves produced by CleaneX have lower variance than those of the regression method (which do not always decrease with k ), and on average our method outperforms the regression, achieving lower prediction errors by 18% (CIFAR), 32% (LFW) and 22% (fMRI).

Original: On the other hand, the KDE method achieves lower variance but high bias in two of three experiments (Figure 3.C), where it is outperformed by CleaneX by 7% (CIFAR) and 73% (fMRI).
Revised: On the other hand, the KDE method achieves lower variance but high bias in two of three experiments (Figure 3.C), where it is outperformed by CleaneX by 7% (CIFAR) and 73% (fMRI).

Original: However, in contrast to the simulation results and its performance on the CIFAR and fMRI datasets, the KDE method achieves exceptionally low bias on the LFW dataset, outperforming our method by 38% .
Revised: However, in contrast to the simulation results and its performance on the CIFAR and fMRI datasets, the KDE method achieves exceptionally low bias on the LFW dataset, outperforming our method by 38% .

Original: Overall, CleaneX produces more reliable results across the experiments
Revised: Overall, CleaneX produces more reliable results across the experiments

Original: [Figure]
Revised: [Figure]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work we presented the reversed ROC and showed its connection to the accuracy of marginal classiﬁers.
Revised: In this work we presented the reversed ROC and showed its connection to the accuracy of marginal classiﬁers.

Original: We used this result to develop a new method for accuracy prediction.
Revised: We used this result to develop a new method for accuracy prediction.

Original: Analysis of the two previous methods for accuracy extrapolation reveals that each of them uses only part of the relevant information for the task.
Revised: Analysis of the two previous methods for accuracy extrapolation reveals that each of them uses only part of the relevant information for the task.

Original: The KDE method estimates C x based only on the scores, ignoring the observed accuracies of the classiﬁer.
Revised: The KDE method estimates C x based only on the scores, ignoring the observed accuracies of the classiﬁer.

Original: Even when the estimates of C x are unbiased, the exponentiation in E
Revised: Even when the estimates of C x are unbiased, the exponentiation in E

Original: x
Revised: x

Original: [ C k − 1 x ] introduces bias.
Revised: [ C k − 1 x ] introduces bias.

Original: Since the estimation is not calibrated using the observed accuracies, the resulting estimations are often biased.
Revised: Since the estimation is not calibrated using the observed accuracies, the resulting estimations are often biased.

Original: As can be seen in Figure 2, the bias is higher for k 1 = 500 than for k 1 = 100 , indicating that this is aggregated when k 1 is small.
Revised: As can be seen in Figure 2, the bias is higher for k 1 = 500 than for k 1 = 100 , indicating that this is aggregated when k 1 is small.

Original: In addition, we found the method to be sensitive to monotone transformations of the scores, such as taking squared-root or logarithm.
Revised: In addition, we found the method to be sensitive to monotone transformations of the scores, such as taking squared-root or logarithm.

Original: In contrast, the non-parametric regression based method uses pre-chosen basis functions to predict the accuracy curves and therefore has limited versatility to capture complicated functional forms.
Revised: In contrast, the non-parametric regression based method uses pre-chosen basis functions to predict the accuracy curves and therefore has limited versatility to capture complicated functional forms.

Original: Since it ignores the values of the classiﬁcation scores, the resulting predicted curves do not necessarily follow the form of E x [ C k − 1 x ] , introducing high variance estimations.
Revised: Since it ignores the values of the classiﬁcation scores, the resulting predicted curves do not necessarily follow the form of E x [ C k − 1 x ] , introducing high variance estimations.

Original: As can be seen in Figure 2, the variance is higher for k 1 = 100 compared to k 1 = 500 , indicating higher variance for small values of k 1 .
Revised: As can be seen in Figure 2, the variance is higher for k 1 = 100 compared to k 1 = 500 , indicating higher variance for small values of k 1 .

Original: This can be expected since the number of basis functions that are used to approximate the discriminability function depends on k 1 .
Revised: This can be expected since the number of basis functions that are used to approximate the discriminability function depends on k 1 .

Original: In CleaneX, we exploit the mechanism of neural networks in a manner that allows us to combine both sources of information with less restriction on the shape of the curve.
Revised: In CleaneX, we exploit the mechanism of neural networks in a manner that allows us to combine both sources of information with less restriction on the shape of the curve.

Original: Since the extrapolation follows the known exponential form using the estimated C x values it is characterized with low variance, and since the result is calibrated by optimizing an objective that depends on the observed accuracies, our method has low bias, and therefore consistently outperforms the previous methods.
Revised: Since the extrapolation follows the known exponential form using the estimated C x values it is characterized with low variance, and since the result is calibrated by optimizing an objective that depends on the observed accuracies, our method has low bias, and therefore consistently outperforms the previous methods.

Original: Comparing the results for k 1 = 500 between different dimensions ( d = 5 in Figure 2 and d = 3 ,in Figure 6) it can be seen that the bias of the KDE based method and the variance of the regression based methods are lower in higher data dimensions.
Revised: Comparing the results for k 1 = 500 between different dimensions ( d = 5 in Figure 2 and d = 3 ,in Figure 6) it can be seen that the bias of the KDE based method and the variance of the regression based methods are lower in higher data dimensions.

Original: Zheng & Benjamini (2016) show that if both
Revised: Zheng & Benjamini (2016) show that if both

Original: X and Y are high dimensional vectors and their joint distribution factorizes, then the classiﬁcation scores of Bayesian classiﬁers (which are marginal) are approximately Gausssian.
Revised: X and Y are high dimensional vectors and their joint distribution factorizes, then the classiﬁcation scores of Bayesian classiﬁers (which are marginal) are approximately Gausssian.

Original: The KDE and the regression based methods use Gaussian kernel and Gaussian-based basis functions respectively, and are perhaps more efﬁcient in estimating approximately Gaussian score distributions.
Revised: The KDE and the regression based methods use Gaussian kernel and Gaussian-based basis functions respectively, and are perhaps more efﬁcient in estimating approximately Gaussian score distributions.

Original: Apparently, scores for real data-sets behave closer to low-dimensional data, where the ﬂexibility of CleaneX (partly due to being a non-parametric method) is an advantage.
Revised: Apparently, scores for real data-sets behave closer to low-dimensional data, where the ﬂexibility of CleaneX (partly due to being a non-parametric method) is an advantage.

Original: A considerable source of noise in the estimation is the selection of the k 1 classes.
Revised: A considerable source of noise in the estimation is the selection of the k 1 classes.

Original: The ¯ A k 1 k curves diverge from E k
Revised: The ¯ A k 1 k curves diverge from E k

Original: [ A ] as the number of classes increases, and therefore it is hard to recover if the initial k 1 classes deviated from the true accuracy.
Revised: [ A ] as the number of classes increases, and therefore it is hard to recover if the initial k 1 classes deviated from the true accuracy.

Original: The effect of choosing different initial subsets can be seen by comparing the grey curves and the orange curves that continue them, for example in Figure 3.
Revised: The effect of choosing different initial subsets can be seen by comparing the grey curves and the orange curves that continue them, for example in Figure 3.

Original: We leave the design of more informative sampling schemes for future work.
Revised: We leave the design of more informative sampling schemes for future work.

Original: To conclude, we found the method we present to be considerably more stable than previous methods.
Revised: To conclude, we found the method we present to be considerably more stable than previous methods.

Original: Therefore, it is now possible for researchers to reliably estimate how well the classiﬁcation system they are developing will perform using representations learned only on a subset of the target class set.
Revised: Therefore, it is now possible for researchers to reliably estimate how well the classiﬁcation system they are developing will perform using representations learned only on a subset of the target class set.

Original: Although our work focuses on marginal classiﬁers, its importance extends beyond this class of algorithms.
Revised: Although our work focuses on marginal classiﬁers, its importance extends beyond this class of algorithms.

Original: First, preliminary results show that our method yields good estimates even when applied to (shallow) non-marginal classiﬁers such as multi-logistic regression.
Revised: First, preliminary results show that our method yields good estimates even when applied to (shallow) non-marginal classiﬁers such as multi-logistic regression.

Original: Moreover, if the representation can adapt as classes are introduced, we expect accuracy to exceed that of a ﬁxed representation.
Revised: Moreover, if the representation can adapt as classes are introduced, we expect accuracy to exceed that of a ﬁxed representation.

Original: We can therefore use our algorithm to measure the degree of adaptation of the representation.
Revised: We can therefore use our algorithm to measure the degree of adaptation of the representation.

Original: Generalization of our method to non marginal classiﬁers is a prominent direction for future work.
Revised: Generalization of our method to non marginal classiﬁers is a prominent direction for future work.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: TBD
Revised: We thank Etam Benger for many fruitful discussions, and Itamar Faran and Charles Zheng for commenting on the manuscript. YS is supported by the Israeli Council For Higher Education DataScience fellowship and the CIDR center at the Hebrew University of Jerusalem.

Original: A P ROOF OF THEOREM
Revised: A P ROOF OF THEOREM

Original: We begin by proving the following simple proposition:
Revised: We begin by proving the following simple proposition:

Original: Proposition 1.
Revised: Proposition 1.

Original: Given x , let
Revised: Given x , let

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: then G 1 = G 2 .
Revised: then G 1 = G 2 .

Original: Proof.
Revised: Proof.

Original: Let G c 1 , G c 2 be the complement sets of G 1 , G 2 , respectively.
Revised: Let G c 1 , G c 2 be the complement sets of G 1 , G 2 , respectively.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: ) and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Since P Y (cid:48) ( S y (cid:48) ( x ) ≥ t ) is decreasing in t we have
Revised: Since P Y (cid:48) ( S y (cid:48) ( x ) ≥ t ) is decreasing in t we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and therefore G 1 = G 2 .
Revised: and therefore G 1 = G 2 .

Original: We now prove Theorem 1, which was presented in Section 3:
Revised: We now prove Theorem 1, which was presented in Section 3:

Original: Proof.
Revised: Proof.

Original: We show that 1 − rROC (1 − u )
Revised: We show that 1 − rROC (1 − u )

Original: =
Revised: =

Original: D ( u ) , and the rest follows immediately according to
Revised: D ( u ) , and the rest follows immediately according to

Original: Theorem 3 of Zheng et al.
Revised: Theorem 3 of Zheng et al.

Original: (see Equation 4):
Revised: (see Equation 4):

Original: [Equation]

Original: [Equation]

Original: Proposition

Original: [Equation]
Revised: [Equation]

Original: where 1 [ · ] denotes the indicator function, and the last equality follows from the assumption that
Revised: where 1 [ · ] denotes the indicator function, and the last equality follows from the assumption that

Original: S y (cid:48)
Revised: S y (cid:48)

Original: ( x ) (cid:54) =
Revised: ( x ) (cid:54) =

Original: S y
Revised: S y

Original: ( x ) almost surely.
Revised: ( x ) almost surely.

Original: From here,
Revised: From here,

Original: [Equation]
Revised: [Equation]

Original: as required.
Revised: as required.

Original: B S IMULATIONS - ADDITIONAL RESULTS
Revised: B S IMULATIONS - ADDITIONAL RESULTS

Original: Figures 4 and 5 show the accuracy curves for which we presented summarized results in Figure 2.
Revised: Figures 4 and 5 show the accuracy curves for which we presented summarized results in Figure 2.

Original: In Figure 6 we provide additional simulation results for data in dimensions d = 3 , 10 .
Revised: In Figure 6 we provide additional simulation results for data in dimensions d = 3 , 10 .

Original: We simulate eight d -dimensional datasets of k 2 = 2000 classes and extrapolate the accuracy from k 1 =classes.
Revised: We simulate eight d -dimensional datasets of k 2 = 2000 classes and extrapolate the accuracy from k 1 =classes.

Original: As in Section 5.1, the datasets are combinations of two distributions of Y and two of
Revised: As in Section 5.1, the datasets are combinations of two distributions of Y and two of

Original: X | Y , in two different levels of classiﬁcation difﬁculty.
Revised: X | Y , in two different levels of classiﬁcation difﬁculty.

Original: We sample the classes y 1 , . . .
Revised: We sample the classes y 1 , . . .

Original: , y k 2 from a multivariate normal distribution N (0 , I ) or a multivariate uniform distribution U ( − 1 , 1) .
Revised: , y k 2 from a multivariate normal distribution N (0 , I ) or a multivariate uniform distribution U ( − 1 , 1) .

Original: We sample r = 10 data points for each class from a multivariate normal distribution N ( y, σ 2 I ) or from
Revised: We sample r = 10 data points for each class from a multivariate normal distribution N ( y, σ 2 I ) or from

Original: Y ∼ N ,
Revised: Y ∼ N ,

Original: X | Y ∼
Revised: X | Y ∼

Original: N σ 2 = 0 .0 .0 .0 .0 .0 .1 .
Revised: N σ 2 = 0 .0 .0 .0 .0 .0 .1 .

Original: A cc u r a c y
Revised: A cc u r a c y

Original: CleaneX
Revised: CleaneX

Original: Regression
Revised: Regression

Original: KDE
Revised: KDE

Original: Y ∼ N ,
Revised: Y ∼ N ,

Original: X | Y ∼
Revised: X | Y ∼

Original: N σ 2 = 0 .0 .0 .0 .0 .0 .1 .
Revised: N σ 2 = 0 .0 .0 .0 .0 .0 .1 .

Original: A cc u r a c y
Revised: A cc u r a c y

Original: Y ∼ N ,
Revised: Y ∼ N ,

Original: X | Y ∼
Revised: X | Y ∼

Original: U σ 2 = 0 .0 .0 .0 .0 .0 .1 .
Revised: U σ 2 = 0 .0 .0 .0 .0 .0 .1 .

Original: A c c u r a c y
Revised: A c c u r a c y

Original: Y ∼ N ,
Revised: Y ∼ N ,

Original: X | Y ∼
Revised: X | Y ∼

Original: U σ 2 = 0 .0 .0 .0 .0 .0 .1 .
Revised: U σ 2 = 0 .0 .0 .0 .0 .0 .1 .

Original: A cc u r a c y
Revised: A cc u r a c y

Original: Y ∼ U ,
Revised: Y ∼ U ,

Original: X | Y ∼
Revised: X | Y ∼

Original: N σ 2 = 0 .0 .0 .0.0 .0 .1 .
Revised: N σ 2 = 0 .0 .0 .0 .

Original: A cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: ∼ U ,

Original: [Figure]
Revised: [Figure]

Original: |

Original: [Figure]
Revised: [Figure]

Original: ∼ N σ 2 = 0 . [Figure]
Revised: r

Original: [Figure]
Revised: [Figure]

Original: Y ∼ N ,
Revised: Y ∼ N ,

Original: X | Y ∼
Revised: X | Y ∼

Original: N σ 2 = 0 .0 .0 .0 .0 .0 .1 .
Revised: N σ 2 = 0 .0 .0 .0 .0 .0 .1 .

Original: A cc u r a c y
Revised: A cc u r a c y

Original: CleaneX
Revised: CleaneX

Original: Regression
Revised: Regression

Original: KDE
Revised: KDE

Original: Y ∼ N ,
Revised: Y ∼ N ,

Original: X | Y ∼
Revised: X | Y ∼

Original: N σ 2 = 0 .0 .0 .0 .0 .0 .1 .
Revised: N σ 2 = 0 .0 .0 .0 .0 .0 .1 .

Original: A cc u r a c y
Revised: A cc u r a c y

Original: Y ∼ N ,
Revised: Y ∼ N ,

Original: X | Y ∼
Revised: X | Y ∼

Original: U σ 2 = 0 .0 .0 .0 .0 .0 .1 .
Revised: U σ 2 = 0 .0 .0 .0 .0 .0 .1 .

Original: A c c u r a c y
Revised: A c c u r a c y

Original: Y ∼ N ,
Revised: Y ∼ N ,

Original: X | Y ∼
Revised: X | Y ∼

Original: U σ 2 = 0 .0 .0 .0 .0 .0 .1 .
Revised: U σ 2 = 0 .0 .0 .0 .0 .0 .1 .

Original: A cc u r a c y
Revised: A cc u r a c y

Original: Y ∼ U ,
Revised: Y ∼ U ,

Original: X | Y ∼
Revised: X | Y ∼

Original: N σ 2 = 0 .0 .0 .0.0 .0 .1 .
Revised: N σ 2 = 0 .0 .0 .0 .

Original: A cc u r a c y

Original: Y ∼ U ,

Original: [Figure]

Original: |

Original: [Figure]
Revised: [Figure]

Original: ∼

Original: N σ 2 = 0 .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: a multivariate uniform distribution U ( y − σ 2 , y + σ 2 ) .
Revised: a multivariate uniform distribution U ( y − σ 2 , y + σ 2 ) .

Original: For d = 3 we set σ 2 to 0.1 for the easier classiﬁcation task and 0.2 for the difﬁcult one.
Revised: For d = 3 we set σ 2 to 0.1 for the easier classiﬁcation task and 0.2 for the difﬁcult one.

Original: For d = 10 , the values of σ 2 are set to 0.6 and 0.9.
Revised: For d = 10 , the values of σ 2 are set to 0.6 and 0.9.

Original: The classiﬁcation scores S y
Revised: The classiﬁcation scores S y

Original: ( x ) are the euclidean distances between x and y .
Revised: ( x ) are the euclidean distances between x and y .

Original: It can be seen that the results for d = 3 are consistent with those for d = 5 .
Revised: It can be seen that the results for d = 3 are consistent with those for d = 5 .

Original: That is, our method outperforms the two competing methods, even with a bigger gap, in seven of the eight simulated datasets.
Revised: That is, our method outperforms the two competing methods, even with a bigger gap, in seven of the eight simulated datasets.

Original: For d = 10 the regression based method achieves compatible results with ours.
Revised: For d = 10 the regression based method achieves compatible results with ours.

Original: As in the other simulations, our method produces less variance than the regression and less bias than KDE.
Revised: As in the other simulations, our method produces less variance than the regression and less bias than KDE.

Original: [Figure]
Revised: [Figure]

Original: C I MPLEMENTATION D ETAILS
Revised: C I MPLEMENTATION DETAILS

Original: All the code in this work was implemented in Python 3.6.
Revised: All the code in this work was implemented in Python 3.6.

Original: For the CleaneX algorithm we used
Revised: For the CleaneX algorithm we used

Original: TensorFlow 1.14; for the regression based method we used the scipy.optimize package with the “Newton-CG” method; kernel density estimation was implemented using the density function from the stats library in R, imported to Python through the rpy2 package.
Revised: TensorFlow 1.14; for the regression based method we used the scipy.optimize package with the “Newton-CG” method; kernel density estimation was implemented using the density function from the stats library in R, imported to Python through the rpy2 package.