Original: S ELECTIVE C LASSIFIER E NSEMBLE
Revised: S ELECTIVE C LASSIFIER E NSEMBLE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Selective classiﬁcation allows a machine learning model to abstain on some hard inputs and thus improve the safety of its predictions.
Revised: Selective classiﬁcation allows a machine learning model to abstain from predicting some hard inputs and thus improve the safety of its predictions.

Original: In this paper, we study the ensemble of selective classiﬁers, i.e. selective classiﬁer ensemble , which combines several weak selective classiﬁers to obtain a more powerful model.
Revised: In this paper, we study the ensemble of selective classiﬁers, i.e. selective classiﬁer ensemble , which combines several weak selective classiﬁers to obtain a more powerful model.

Original: We prove that under some assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.
Revised: We prove that under some assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.

Original: This is nontrivial since the selective risk is a non-convex function of the model prediction.
Revised: The proof is nontrivial since the selective risk is a non-convex function of the model prediction.

Original: The assumptions and the theoretical result are supported by systematic experiments on both computer vision and natural language processing tasks.
Revised: The assumptions and the theoretical result are supported by systematic experiments on both computer vision and natural language processing tasks.

Original: A surprising empirical result is that a simple selective classiﬁer ensemble, namely, the ensemble model with maximum probability as conﬁdence , is the state-of-the-art selective classiﬁer.
Revised: A surprising empirical result is that a simple selective classiﬁer ensemble, namely, the ensemble model with maximum probability as conﬁdence , is the state-of-the-art selective classiﬁer.

Original: For instance, on CIFAR-10, using the same VGG-16 backbone model, this ensemble reduces the AURC (Area Under Risk-Coverage Curve) by about 24%, relative to the previous state-of-the-art method.
Revised: For instance, on CIFAR-10, using the same VGG-16 backbone model, this ensemble reduces the AURC (Area Under Risk-Coverage Curve) by about 24%, relative to the previous state-of-the-art method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Although recent years have witnessed the broad applications of deep learning models, their securities have not been fully guaranteed, which gives rise to the study of selective classiﬁcation.
Revised: Although recent years have witnessed the broad applications of deep learning models, their securities have not been fully guaranteed, which gives rise to the study of selective classiﬁcation.

Original: For any given deep learning classiﬁer, there might be inputs that the model is not able to classify in practical applications, for which the model might make unpredictable errors.
Revised: For any given deep learning classiﬁer, there might be inputs that the model is not able to classify in practical applications, for which the model might make unpredictable errors.

Original: To prevent this kind of error, we need to delimit theapplication scope of the deep learning classiﬁer accurately.
Revised: To prevent this kind of error, we must accurately delimit the deep learning classiﬁer’s application scope.

Original: This  gives rise to the study of selective classiﬁcation that learns a selective classiﬁer ( f, g ) , where f is a conventional classiﬁer and g is a selective function that decides whether the selective classiﬁer should abstain from prediction.
Revised: This need gives rise to the study of selective classiﬁcation that learns a selective classiﬁer ( f, g ) , where f is a conventional classiﬁer, and g is a selective function that decides whether the selective classiﬁer should abstain from prediction.

Original: Since the classiﬁer is well studied, the study of selective classiﬁcation focuses on the design of the selective function.
Revised: Since the classiﬁer is well studied, the study of selective classiﬁcation focuses on the design of the selective function.

Original: A standard approach to designing the selective function is to design a conﬁdence score function with a threshold, and several conﬁdence score functions have been developed.
Revised: A standard approach to designing the selective function is to design a conﬁdence score function with a threshold, and several conﬁdence score functions have been developed.

Original: A simple conﬁdence score function is the maximum predictive probability of the classiﬁer (Hendrycks & Gimpel, 2017).
Revised: A simple conﬁdence score function is the maximum predictive probability of the classiﬁer (Hendrycks & Gimpel, 2017).

Original: More advanced methods modifyeither the model architecture (Geifman & El-Yaniv, 2019) or the loss function (Liu et al., 2019; Huang et al., 2020) of the classiﬁer to train the conﬁdence score function and the classiﬁer simultaneously.
Revised: More advanced methods modify the model architecture (Geifman & El-Yaniv, 2019) or the loss function (Liu et al., 2019; Huang et al., 2020) of the classiﬁer to train the conﬁdence score function and the classiﬁer simultaneously.

Original: For example, Deep Gambler (Liu et al., 2019) regards the selective classiﬁcation problem as gambling and proposes a novel loss function to train the classiﬁer and the conﬁdence score function.
Revised: For example, Deep Gambler (Liu et al., 2019) regards the selective classiﬁcation problem as gambling and proposes a novel loss function to train the classiﬁer and the conﬁdence score function.

Original: Although there are various individual models for the selective classiﬁer, there has been no systematic study of the ensemble method in selective classiﬁcation.
Revised: Although there are various individual models for the selective classiﬁer, there has been no systematic study of the ensemble method in selective classiﬁcation.

Original: It is well known that the ensemble method, which combines the individual models to obtain a more powerful model, can improve  predictive performances of machine learning models (see Zhou (2012) for a review), but there has been no systematic study of ensemble methods in selective classiﬁcation.
Revised: It is well known that the ensemble method, which combines the individual models to obtain a more powerful model, can improve the predictive performances of machine learning models (see Zhou (2012) for a review), but only a particular selective classiﬁer ensemble, the ensemble of Softmax

Original: Although the ensemble of Softmax Response (Hendrycks & Gimpel, 2017) has been empirically studied by Lakshminarayanan et al.
Revised: Response (Hendrycks & Gimpel, 2017), has been empirically studied by Lakshminarayanan et al.

Original: (2017), ensembles of other kinds of selective classiﬁers and the theoretical foundation of the ensemble in selective classiﬁcation have not been studied yet.
Revised: Ensembles of other kinds of selective classiﬁers, and the theoretical foundation of the ensemble in selective classiﬁcation have not been studied yet.

Original: In this paper, we ﬁrst demonstrate the theoretical foundation of the ensemble on selective classiﬁers, that is, with some assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.
Revised: In this paper, we ﬁrst demonstrate the theoretical foundation of the ensemble on selective classiﬁers, that is, with some assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.

Original: This is nontrivial since the selective risk (with the 0/1 loss) are nonconvex.
Revised: The proof is nontrivial since the selective risk (with the 0/1 loss) are non-convex.

Original: Second, we show the experimental results of the  performanceof the ensemble in selective classiﬁcation.
Revised: Second, we show the experimental results of the ensemble’s performance in selective classiﬁcation.

Original: The contributions of this paper are summarized as follows.
Revised: The contributions of this paper are summarized as follows.

Original: • We are the ﬁrst to theoretically demonstrate that based on several reasonable assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.
Revised: • We are the ﬁrst to theoretically demonstrate that based on several reasonable assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.

Original: We verify this by systematic experiments on the tasks of image classiﬁcation and text classiﬁcation.
Revised: We verify this by systematic experiments on the tasks of image classiﬁcation and text classiﬁcation.

Original: • We show a surprising experimental result that two simple methods, the SR ensemble and the Reg-curr ensemble, which can be summarized as the ensemble model with maximum probability as conﬁdence , are the state-of-the-art selective classiﬁers.
Revised: • We show a surprising experimental result that two simple methods, the SR ensemble and the Reg-curr ensemble, which can be summarized as the ensemble model with maximum probability as conﬁdence , are the state-of-the-art selective classiﬁers.

Original:  P ROBLEM F ORMULATION OF S ELECTIVE C LASSIFICATION
Revised:  P ROBLEM F ORMULATION OF S ELECTIVE C LASSIFICATION

Original: A selective classiﬁer is composed of a standard classiﬁer and a selective function.
Revised: A selective classiﬁer is composed of a standard classiﬁer and a selective function.

Original: Considering a standard classiﬁcation problem, X is a feature space, Y = { 1 , 2 , ..., K } is a ﬁnite label set, and a classiﬁer f is a function f : X → Y .
Revised: Considering a standard classiﬁcation problem, X is a feature space, Y = { 1 , 2 , ..., K } is a ﬁnite label set, and a classiﬁer f is a function f : X → Y .

Original: A labeled dataset D = { ( x i , y i ) } Ni =0 ⊆
Revised: A labeled dataset D = { ( x i , y i ) } Ni =0 ⊆

Original: X × Y is sampled from a distribution p X , Y .
Revised: X × Y is sampled from a distribution p X , Y .

Original: Our goal is to learn a selective classiﬁer where f is a standard classiﬁer and g : X → { 0 , 1 } is a selective function that estimates the correctness of f ’s prediction.
Revised: Our goal is to learn a selective classiﬁer where f is a standard classiﬁer and g : X → { 0 , 1 } is a selective function that estimates the correctness of f ’s prediction.

Original: Given input x , the output of selective classiﬁer ( f, g ) is ( f, g )( x ) = (cid:26) f ( x ) , if g ( x ) =
Revised: Given input x , the output of selective classiﬁer ( f, g ) is ( f, g )( x ) = (cid:26) f ( x ) , if g ( x ) =

Original: Abstain , if g ( x ) = 0 .
Revised: Abstain , if g ( x ) = 0 .

Original: Usually, g is realized by a conﬁdence score ˆ κ : X → R + with a threshold τ (Geifman & El-Yaniv, 2017), namely g ( x ) =
Revised: Usually, g is realized by a conﬁdence score ˆ κ : X → R + with a threshold τ (Geifman & El-Yaniv, 2017), namely g ( x ) =

Original: I { ˆ κ ( x ) > τ } , (2) where I is the indicator function.
Revised: I { ˆ κ ( x ) > τ } , (2) where I is the indicator function.

Original: Coverage and selective risk are two basic evaluation metrics of selective classiﬁers, and the goal of selective classiﬁers is to minimize the selective risk fora target coverage.
Revised: Coverage and selective risk are two basic evaluation metrics of selective classiﬁers, and the goal of selective classiﬁers is to minimize the selective risk for target coverage.

Original: The coverage of ( f, g ) is deﬁned to be the probability of ( f, g ) not abstaining from prediction (Geifman & El-Yaniv, 2017), i.e.
Revised: The coverage of ( f, g ) is deﬁned to be the probability of ( f, g ) not abstaining from prediction (Geifman & El-Yaniv, 2017), i.e.

Original: [Equation]
Revised: [Equation]

Original: where p ( x ) is the probability density function of input x .
Revised: where p ( x ) is the probability density function of input x .

Original: The selective risk (Geifman & El-Yaniv, 2017) of ( f, g ) is
Revised: The selective risk (Geifman & El-Yaniv, 2017) of ( f, g ) is

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) :
Revised: where (cid:96) :

Original: Y × Y → R + is a given loss function.
Revised: Y × Y → R + is a given loss function.

Original: Usually, (cid:96) is the 0/1 loss (Geifman & El-Yaniv, 2017; 2019;
Revised: Usually, (cid:96) is the 0/1 loss (Geifman & El-Yaniv, 2017; 2019;

Original: Liu et al., 2019; Huang et al., 2020).
Revised: Liu et al., 2019; Huang et al., 2020).

Original: Based on these deﬁnitions, the objective of selective classiﬁers is formalized as
Revised: Based on these deﬁnitions, the objective of selective classiﬁers is formalized as

Original: [Equation]
Revised: [Equation]

Original: where c target is a given target coverage.
Revised: where c target is a given target coverage.

Original: When the selective function g is developed as (2), the conﬁdence threshold τ controls the tradeoff between coverage and selective risk.
Revised: When the selective function g is developed as (2), the conﬁdence threshold τ controls the tradeoff between coverage and selective risk.

Original: With different values of τ , ( f, g ) has different pairs of coverage and selective risk ( φ ( f, g ; τ ) , R ( f, g ; τ )) , which forms the risk-coverage curve (Geifman & El-Yaniv, 2017) of ( f, g ) .
Revised: With different values of τ , ( f, g ) has different pairs of coverage and selective risk ( φ ( f, g ; τ ) , R ( f, g ; τ )) , which forms the risk-coverage curve (Geifman & El-Yaniv, 2017) of ( f, g ) .

Original: The risk-coverage curve speciﬁes the entire performance proﬁle of a selective classiﬁer, and it is easy to see that the selective classiﬁer with a lower risk-coverage curve is better.
Revised: The risk-coverage curve speciﬁes the entire performance proﬁle of a selective classiﬁer, and it is easy to see that the selective classiﬁer with a lower risk-coverage curve is better.

Original: To evaluate selective classiﬁers more concisely, the area under the risk-coverage curve ( AURC ) is introduced as a metric of selective classiﬁers (Xin et al., 2021), and the selective classiﬁer with a lower AURC is better.
Revised: To evaluate selective classiﬁers more concisely, the area under the risk-coverage curve ( AURC ) is introduced as a metric of selective classiﬁers (Xin et al., 2021), and the selective classiﬁer with a lower AURC is better.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Here, we summarize the previous studies on selective classiﬁcation and ensemble methods.
Revised: Here, we summarize the previous studies on selective classiﬁcation and ensemble methods.

Original: We also discuss the difference between selective classiﬁcation and out-of-distribution detection.
Revised: We also discuss the difference between selective classiﬁcation and out-of-distribution detection.

Original: 3. S ELECTIVE C LASSIFICATION
Revised: 3. S ELECTIVE C LASSIFICATION

Original: The critical problem of designing a selective classiﬁer is to design its selective function, and there are two types ofthe selective function g .
Revised: The critical problem of designing a selective classiﬁer is to design its selective function, and there are two types of selective function g .

Original: One is the implicit selective function , which is derived from the classiﬁer.
Revised: One is the implicit selective function , which is derived from the classiﬁer.

Original: The other is the explicit selective function ,which is a neural network trained with the classiﬁer f simultaneously.
Revised: The other is the explicit selective function , a neural network trained with the classiﬁer f simultaneously.

Original: Previous works for selective classiﬁers are listed as follows.
Revised: Previous works for selective classiﬁers are listed as follows.

Original: Selective classiﬁers with implicit selective functions include SR (Softmax Response) (Hendrycks & Gimpel, 2017), MC-Dropout (Monte Carlo-Dropout) (Gal & Ghahramani, 2016), and Reg-curr (Xin et al., 2021).
Revised: Selective classiﬁers with implicit selective functions include SR (Softmax Response) (Hendrycks & Gimpel, 2017), MC-Dropout (Monte Carlo-Dropout) (Gal & Ghahramani, 2016), and Reg-curr (Xin et al., 2021).

Original: In SR, the selective classiﬁer is a vanilla classiﬁer with its maximum predictive probability as conﬁdence (i.e. the maximumof the output of the softmax layer).
Revised: In SR, the selective classiﬁer is a vanilla classiﬁer with its maximum predictive probability as conﬁdence (i.e., the maximum output of the softmax layer).

Original: In MC-Dropout, it enables the dropout layer of the classiﬁer and runs multiple feed-forward iterations at inference time to obtain the variance of the maximum probability output of f ’s softmax layer, whose negative value is used as the conﬁdence score.
Revised: In MC-Dropout, it enables the dropout layer of the classiﬁer and runs multiple feed-forward iterations at inference time to obtain the variance of the maximum probability output of f ’s softmax layer, whose negative value is used as the conﬁdence score.

Original: Reg-curr behaves the same as the SR at inference time but uses an RPP-based regularizer at training time, where RPP (Reversed Pair Proportion) (Xin et al., 2021) is the proportion of reversed pairs of conﬁdence scores.
Revised: Reg-curr behaves the same as the SR at inference time but uses an RPPbased regularizer at training time, where RPP (Reversed Pair Proportion) (Xin et al., 2021) is the proportion of reversed pairs of conﬁdence scores.

Original: Selective classiﬁers with explicit selective functions include SN (SelectiveNet) (Geifman & ElYaniv, 2019), Gambler (Deep Gambler)
Revised: Selective classiﬁers with explicit selective functions include SN (SelectiveNet) (Geifman & ElYaniv, 2019), Gambler (Deep Gambler)

Original: (Liu et al., 2019), and SAT (Self-Adaptive Training) (Huang et al., 2020).
Revised: (Liu et al., 2019), and SAT (Self-Adaptive Training) (Huang et al., 2020).

Original: SN is a neural network that combines f and g , where f and g share convolutional layers and have their separate fully-connected layers.
Revised: SN is a neural network that combines f and g , where f and g share convolutional layers and have their separate fully-connected layers.

Original: The loss function of the model is the selective risk with some regularizers.
Revised: The loss function of the model is the selective risk with some regularizers.

Original: A hyperparameter c is needed to specify the target coverage.
Revised: A hyperparameter c is needed to specify the target coverage.

Original: Gambler adds the abstention option to the classiﬁer as an extra class, that is, for a given input x , the predictive probability of the extra class is the conﬁdence of abstention, i.e, 1 − ˆ κ ( x ) .
Revised: Gambler adds the abstention option to the classiﬁer as an extra class, that is, for a given input x , the predictive probability of the extra class is the conﬁdence of abstention, i.e., 1 − ˆ κ ( x ) .

Original: At training time, it regards the selective classiﬁcation problem as gambling and is trained to maximize the gambling reward.
Revised: At training time, it regards the selective classiﬁcation problem as gambling and is trained to maximize the gambling reward.

Original: Similar to Gambler, SAT adds the abstention option to the classiﬁer as an extra class.
Revised: Similar to Gambler, SAT adds the abstention option to the classiﬁer as an extra class.

Original: However, SAT has a different training procedure, which is trained with a soft label that tells the modelon which sample to abstain.
Revised: However, SAT has a different training procedure, which is trained with a soft label that tells the model which sample to reject.

Original: SAT is the previous state-of-the-art selective classiﬁer for image classiﬁcation tasks.
Revised: SAT is the previous state-of-the-art selective classiﬁer for image classiﬁcation tasks.

Original: 3. R ANDOMIZATION - BASED E NSEMBLE
Revised: 3. R ANDOMIZATION - BASED E NSEMBLE

Original: In the randomization-based ensemble method, each member model is trained independently with the same architecture and training procedure but with different randomization seeds for the random initialization of parameters and the random shufﬂing of training data for each training epoch.
Revised: In the randomization-based ensemble method, each member model is trained independently with the same architecture and training procedure but with different randomization seeds for the random initialization of parameters and the random shufﬂing of training data for each training epoch.

Original: At inference time, the predictive probability of the ensemble for each class is the average of those of member models.
Revised: At inference time, the predictive probability of the ensemble for each class is the average of those of member models.

Original: Lakshminarayanan et al.
Revised: Lakshminarayanan et al.

Original: (2017) applies this ensemble method to deep neural networks (Deep Ensemble) and has achieved state-of-the-art performance in uncertainty estimation.
Revised: (2017) applies this ensemble method to deep neural networks (Deep Ensemble) and has achieved state-of-the-art performance in uncertainty estimation.

Original: The difference between Lakshminarayanan et al.
Revised: The difference between Lakshminarayanan et al.

Original: and our work is that they only empirically study the ensemble of vanilla classiﬁers (the SR ensemble), but we not only empirically study ensembles of multiple outstanding selective classiﬁers but also provide the theoretical foundation of the ensemble in selective classiﬁcation.
Revised: and our work is that they only empirically study the ensemble of vanilla classiﬁers (the SR ensemble), but we not only empirically study ensembles of multiple outstanding selective classiﬁers but also provide the theoretical foundation of the ensemble in selective classiﬁcation.

Original: As for the theoretical works, Krogh & Vedelsby (1994) proposes the error-ambiguity decomposition to explain the better performance of the randomizationbased ensemble in regression tasks.
Revised: As for the theoretical works, Krogh & Vedelsby (1994) proposes the error-ambiguity decomposition to explain the better performance of the randomizationbased ensemble in regression tasks.

Original: However, for classiﬁcation tasks, there is no such simple and elegant analysis, since the evaluation metrics are non-convex (Zhou, 2021).
Revised: However, for classiﬁcation tasks, there is no such simple and elegant analysis, since the evaluation metrics are non-convex (Zhou, 2021).

Original: Thus, the corresponding analysis for classiﬁcation tasks needs additional assumptions, e.g. unbiased, uncorrelated, and identically distributed estimation errors for the posterior probability distribution (Tumer & Ghosh, 1996;
Revised: Thus, the corresponding analysis for classiﬁcation tasks needs additional assumptions, e.g., unbiased, uncorrelated, and identically distributed estimation errors for the posterior probability distribution (Tumer & Ghosh, 1996;

Original: Fumera & Roli, 2005).
Revised: Fumera & Roli, 2005).

Original: Nevertheless, these assumptions are impractical (Fumera & Roli, 2005).
Revised: Nevertheless, these assumptions are impractical (Fumera & Roli, 2005).

Original: As far as we know, there is no systematic study of the ensemble in the context of selective classiﬁcation.
Revised: As far as we know, there is no systematic study of the ensemble in the context of selective classiﬁcation.

Original: 3. O UT - OF -D ISTRIBUTION D ETECTION
Revised: 3. O UT - OF -D ISTRIBUTION D ETECTION

Original: A related topic of selective classiﬁcation is out-of-distribution (OOD) detection (Lakshminarayanan et al., 2017))
Revised: A related topic of selective classiﬁcation is out-of-distribution (OOD) detection (Lakshminarayanan et al., 2017))

Original: (also called as open set recognition (Scheirer et al., 2012), or novelty detection (Sch¨olkopf et al., 2001)), which detects samples different greatly from a given dataset, i.e. OOD samples.
Revised: (also called as open set recognition (Scheirer et al., 2012), or novelty detection (Sch ¨ olkopf et al., 2001)), which detects samples that differ signiﬁcantly from a given dataset, i.e., OOD samples.

Original: The essential difference between selective classiﬁcation and OOD detection lies in their different goals.
Revised: The essential difference between selective classiﬁcation and OOD detection lies in their different goals.

Original: The goal of the former is to detect samples where the classiﬁer predicts incorrectly, which depends on both the classiﬁer and samples, while that of the latter is to detect samples different greatly from a given dataset, which depends on samples only.
Revised: The goal of the former is to detect samples where the classiﬁer predicts incorrectly, which depends on both the classiﬁer and samples, while that of the latter is to detect samples that differ signiﬁcantly from a given dataset, which depends on samples only.

Original: In addition, at present, selective classiﬁcation assumes that test data and training data are sampled from the same distribution (Geifman & El-Yaniv, 2017), instead of using OOD test data as OOD detection.
Revised: In addition, at present, selective classiﬁcation assumes that test data and training data are sampled from the same distribution (Geifman & El-Yaniv, 2017), instead of using OOD test data as OOD detection.

Original: Thus, selective classiﬁcation and OOD detection are complementary in preventing error predictions of machine learning models, as Figure 3 shows.
Revised: Thus, selective classiﬁcation and OOD detection are complementary in preventing error predictions of machine learning models, as Figure 3 shows.

Original:  M ETHOD
Revised:  M ETHOD

Original: With the randomization-based ensemble method, we propose the selective classiﬁer ensemble .
Revised: With the randomization-based ensemble method, we propose the selective classiﬁer ensemble .

Original: The basic idea is that each predictive probability (as well as the conﬁdence score in the case of explicit selective functions) of the ensemble should be the average of those of the member models.
Revised: The basic idea is that each predictive probability (as well as the conﬁdence score in the case of explicit selective functions) of the ensemble should be the average of those of the member models.

Original: Formally, we assume that for an input sample x , a classiﬁer f at ﬁrst provides the predictive probability distribution ˆ π θ =
Revised: Formally, we assume that for an input sample x , a classiﬁer f at ﬁrst provides the predictive probability distribution ˆ π θ =

Original: (ˆ π 1 θ , · · · , ˆ π Kθ ) and then makes prediction f
Revised: (ˆ π 1 θ , · · · , ˆ π Kθ ) and then makes prediction f

Original: ( x ; θ ) = arg max 1 ≤ k ≤ K ˆ π kθ ( x ) , where
Revised: ( x ; θ ) = arg max 1 ≤ k ≤ K ˆ π kθ ( x ) , where

Original: K is the number of classes, θ denotes the parameters of f , and ˆ π kθ ( x ) is the predictive probability for class k (the superscript is not an exponent).
Revised: K is the number of classes, θ denotes the parameters of f , and ˆ π kθ ( x ) is the predictive probability for class k (the superscript is not an exponent).

Original: Then, the predictive probability distribution of the ensemble classiﬁer of M member models is
Revised: Then, the predictive probability distribution of the ensemble classiﬁer of M member models is

Original: [Equation]
Revised: [Equation]

Original: The ensemble of the selective function is deﬁned as follows.
Revised: The ensemble of the selective function is deﬁned as follows.

Original: For implicit selective functions (e.g.
Revised: For implicit selective functions (e.g.

Original: SR), to keep the ensemble the same kind of selective classiﬁer as the individual model (for example, the ensemble of SR should still be an SR model), the conﬁdence score of the ensemble is derived from ˆ π ens in the same way as the individual model.
Revised: SR), to keep the ensemble the same kind of selective classiﬁer as the individual model (for example, the ensemble of SR should still be an SR model), the conﬁdence score of the ensemble is derived from ˆ π ens in the same way as the individual model.

Original: For example, the conﬁdence score of the SR ensemble is
Revised: For example, the conﬁdence score of the SR ensemble is

Original: [Equation]
Revised: [Equation]

Original: For explicit selective functions (e.g. SAT), the conﬁdence score of the ensemble is the average of those of member models,
Revised: For explicit selective functions (e.g. SAT), the conﬁdence score of the ensemble is the average of those of member models,

Original: [Equation]
Revised: [Equation]

Original:  T HEORETICAL A NALYSIS OF S ELECTIVE C LASSIFIER E NSEMBLE
Revised:  T HEORETICAL A NALYSIS OF S ELECTIVE C LASSIFIER E NSEMBLE

Original: In this section, we analyze the selective risk (with the 0/1 loss) of the ensemble of a simple selective classiﬁer, the SR ensemble (see Section 4 for its deﬁnition).
Revised: In this section, we analyze the selective risk (with the 0/1 loss) of the ensemble of a simple selective classiﬁer, the SR ensemble (see Section 4 for its deﬁnition).

Original: If the selective risk is a convex function of the predictive probability distribution, then according to the deﬁnition of the convex function, (6) implies that the selective risk of the ensemble is less than or equal to that of the individual model.
Revised: If the selective risk is a convex function of the predictive probability distribution, then according to the deﬁnition of the convex function, (6) implies that the selective risk of the ensemble is less than or equal to that of the individual model.

Original: However, the selective risk is non-convex, because the 0/1 loss is a step function.
Revised: However, the selective risk is non-convex because the 0/1 loss is a step function.

Original: Thus, the analysis is not easy.
Revised: Thus, the analysis is not easy.

Original: To prove a lower selective risk of the ensemble, we need some assumptions.
Revised: We need some assumptions to prove a lower selective risk of the ensemble.

Original: We introduce the assumptions in Section 5.1 (veriﬁed in Section 6.1) and the theoretical results in Section 5.2.
Revised: We introduce the assumptions in Section 5.1 (veriﬁed in Section 6.1) and the theoretical results in Section 5.2.

Original: The analysis for the other selective classiﬁers is left for future study.
Revised: The analysis for the other selective classiﬁers is left for future study.

Original: 5. A SSUMPTIONS
Revised: 5. A SSUMPTIONS

Original: Given an SR ensemble with M ( M > 1) members, we assume that there are samples on which all member models provide almost the same predictive probability distributions.
Revised: Given an SR ensemble with M ( M > 1) members, we assume that there are samples on which all member models provide almost the same predictive probability distributions.

Original: Furthermore, we idealize them as deﬁnite samples , for which all member models provide exactly the same predictive probability distribution.
Revised: Furthermore, we idealize them as deﬁnite samples , for which all member models provide precisely the same predictive probability distribution.

Original: Then, the restof samples are referred to as ambiguous samples .
Revised: Then, the rest samples are referred to as ambiguous samples .

Original: Let D be the event that the input sample is deﬁnite and A be the event that the input sample is ambiguous.
Revised: Let D be the event that the input sample is deﬁnite and A be the event that the input sample is ambiguous.

Original: Consideringthe case that the input sample is randomly drawn from a dataset, the predictive probability for class k (1 ≤ k ≤ K ), and the conﬁdence of the SR model are random variables.
Revised: Considering that the input sample is randomly drawn from a dataset, the predictive probability for class k (1 ≤ k ≤ K ) and the conﬁdence of the SR model are random variables.

Original: We denote these random variables as Π k and C respectively and use π k and κ to denote their values respectively.
Revised: We denote these random variables as Π k and C respectively and use π k and κ to denote their values respectively.

Original: Generally, for any continuous random variable Z, p Z denotes the probability density function ( PDF ) of Z, and for any real variable z , z → 1 − denotes that z approaches 1 from the left.
Revised: Generally, for any continuous random variable Z, p Z denotes the probability density function ( PDF ) of Z, and for any real variable z , z → 1 − denotes that z approaches 1 from the left.

Original: Based on the idealization and notations above, we introduce the following assumptions.
Revised: Based on the idealization and notations above, we introduce the following assumptions.

Original: Assumption 1.
Revised: Assumption 1.

Original: For any individual SR model, with its conﬁdence score denoted as C, we have
Revised: For any individual SR model, with its conﬁdence score denoted as C, we have

Original: [Equation]
Revised: [Equation]

Original: where Err is the event that the model makes an error prediction.
Revised: where Err is the event that the model makes an error prediction.

Original: In Assumption 1, Pr( Err | A, C ≥ τ ) is the selective risk of the individual model with a conﬁdence threshold of τ for ambiguous samples, and Pr( Err | D, C ≥ τ ) is that for deﬁnite samples.
Revised: In Assumption 1, Pr( Err | A, C ≥ τ ) is the selective risk of the individual model with a conﬁdence threshold of τ for ambiguous samples, and Pr( Err | D, C ≥ τ ) is that for deﬁnite samples.

Original: The motivation of Assumption 1 is that ambiguous samples seem more difﬁcult to classify than deﬁnite samples.
Revised: The motivation for Assumption 1 is that ambiguous samples seem more difﬁcult to classify than deﬁnite samples.

Original: Assumption 2.
Revised: Assumption 2.

Original: For any individual SR model, both lim κ → 1 − p C ( κ | D ) and lim κ → 1 − p C
Revised: For any individual SR model, both lim κ → 1 − p C ( κ | D ) and lim κ → 1 − p C

Original: ( κ | A ) exist and are non-zero, i.e. positive, where C is the conﬁdence score of the SR model.
Revised: ( κ | A ) exist and are non-zero, i.e., positive, where C is the conﬁdence score of the SR model.

Original: Assumption 2 claims that for both ambiguous samples and deﬁnite samples, when the conﬁdence score of the individual model approaches 1, its PDF approaches a non-zero value.
Revised: Assumption 2 claims that for both ambiguous samples and deﬁnite samples, when the conﬁdence score of the individual model approaches 1, its PDF approaches a non-zero value.

Original: This means individual model might assign high conﬁdence to ambiguous sampleswith a considerable probability, which might lead to the overconﬁdent problem, that is, the model assign high conﬁdence to error predictions. In Section 5.2, we prove that the ensemble avoids this problem and thus has a better performance than the individual model.
Revised: In other words, the individual model is not modest over both ambiguous samples and deﬁnite samples.

Original: Assumption 3.
Revised: Assumption 3.

Original: For any SR ensemble of M member models, let Π ki be the predictive probability of the i -th member model for class k .
Revised: For any SR ensemble of M member models, let Π ki be the predictive probability of the i -th member model for class k .

Original: Then the joint probability density function p Π k 1 , Π k 2 ,..., Π k M ( ·| A ) , ∀ k∈ { 1 ,  . .
Revised: Then ∀ k ∈ { 1 , . . , K } , p Π k 1 , Π k 2 ,..., Π kM ( ·| A ) , the joint probability density function of Π k 1 , Π k 2 , . . .

Original: , K } , is bounded. 
Revised: , Π kM given the input sample being ambiguous, is bounded. Assumption 3 is related to the deﬁnition of ambiguous samples and can be understood via the following example. Consider an ensemble with two members θ 1 and θ 2 .

Original: Assumption 3 guarantees that the joint distribution of Π k 1 , Π k 2 , . . , Π kM for ambiguous samples is not so extreme as that for deﬁnite samples.
Revised: Let their predictive probability for class k be Π k 1 and Π k 2 , k ∈ { 1 , 2 , ..., K } , respectively. On deﬁnite samples, both θ 1 and θ 2 provide the same predictive probability distribution ( Π k 1 = Π k 2 , ∀ k ∈ { 1 , 2 , ..., K } ).

Original: For deﬁnite samples, given the predictive probability distribution of one member model (denoted as π 0 ), the predictive probability distributions of other members are certain, so the joint probability density function goes inﬁnite at ( π 0 , ..., π 0 )  . 
Revised: Thus, for all k , p Π k 1 , Π k 2 ( ·| D ) , the joint distribution of Π k 1 and Π k 2 given the input samples being deﬁnite, collapses to { ( λ, λ ) | λ ∈ [0 , 1] } . In other words [Equation] On the contrary, ambiguous samples do not have such property.

Revised: We intensify this by Assumption 3 to provide a good analytical property of ambiguous samples.

Revised: Furthermore, Assumption 3 reﬂects the diversity of the ensemble over ambiguous samples.

Revised: Still consider the example above.

Revised: If the predictions of θ 1 and θ 2 are sure to coincide, i.e., the ensemble model has no diversity, then the PDF of Π k 1 and Π k 2 is unbounded.

Revised: Conversely, if the PDF of Π k 1 and Π k 2 is bounded, then the predictions of the member models are diverse.

Original:  Assumption 3 forbids such extreme case for ambiguous samples.
Revised: Thus, Assumption 3 provides the diversity of the ensemble over ambiguous samples.

Revised: It is well known that the randomization-based ensemble has diversity (Zhou, 2012).

Revised: Since the ensemble does not have diversity over deﬁnite samples, it must have diversity over ambiguous sample.

Revised: Thus, we do not provide experimental veriﬁcation for Assumption 3.

Original: 5. A NALYSIS R ESULTS
Revised: 5. A NALYSIS R ESULTS

Original: With the assumptions above, we obtain Proposition 1 and Theorem 2 (see Appendix B for details of proof).
Revised: With the assumptions above, we derive Theorem 2 (see Appendix B for proof details), which shows that the selective risk of the ensemble is lower than that of the individual model under a range of coverage. The intuition of its proof is as follows. According to Assumption 1, the individual model is not modest over ambiguous samples. On the contrary, based on Assumption 3, we prove that the ensemble is modest over ambiguous samples (Proposition 1).

Revised: In addition, both the individual model and the ensemble are not modest over deﬁnite samples (due to Assumption 2 and the deﬁnition of deﬁnite samples).

Original: Proposition 1 provides the key insight into the better performance of the ensemble, which shows that  the ensemble does not assign high conﬁdence to  ambiguous samples and thus overcomes the problem of the individual model.
Revised: Thus, considering that the classiﬁer’s error rate over ambiguous samples is higher than that over deﬁnite samples when conﬁdence approaches 1 (Assumption 1), the individual model suffers more wrong but conﬁdent predictions that come with ambiguous samples than the ensemble. Therefore, the selective risk of the individual model is higher than that of the ensemble (Theorem 2). In a word, the intuition is that because the ensemble avoids to be overconﬁdent over ambiguous samples, the ensemble has a lower selective risk. Before Theorem 2, we discuss Proposition 1, which provides critical insight into the better performance of the ensemble.

Original: Proposition 1.
Revised: Proposition 1.

Original: If Assumption 3 holds, then lim κ ens → 1 − p C ens ( κ ens | A )
Revised: If Assumption 3 holds, then lim κ ens → 1 − p C ens ( κ ens | A )

Original: = 0 , where C ens is the conﬁdence score of the ensemble.
Revised: = 0 , where C ens is the conﬁdence score of the ensemble.

Original: Proposition 1 suggests that given the input sample being ambiguous, when the conﬁdence of the ensemble approaches one, its PDF approaches zero, so the ensemble does not provide high (close to one) conﬁdence for ambiguous samples in a probabilistic sense.
Revised: Proposition 1 suggests that given the input sample being ambiguous, when the conﬁdence of the ensemble approaches one, its PDF approaches zero. In other words, the ensemble is modest over ambiguous samples.

Original: Based on this, Theorem 2 completes the proof of the better selective classiﬁcation performance of the SR ensemble  than an SR individual model.
Revised: Based on this proposition, we prove that the SR ensemble has a lower selective risk than an SR individual model under a range of coverage.

Original: Theorem 2.
Revised: Theorem 2.

Original: If Assumption 1-3 holds, then  ∃ φ 0 ∈ (0 , 1), such that ∀ φ ∈ ( [Equation]
Revised: If Assumption 1-3 holds, then for any individual SR model and any SR ensemble, ∃ φ 0 ∈ (0 , 1) such that ∀ φ ∈ (0 , φ 0 ) ,

Original: [Equation]
Revised: R ens ( φ ) <

Original:  where R ens ( φ ) and R ind ( φ ) are the selective risksgiven the coverage φ of the SR ensemble and the individual SR model respectively.
Revised: R ind ( φ ) , (9) where R ens ( φ ) and R ind ( φ ) are the selective risks of the SR ensemble and the individual SR model under coverage φ , respectively 1 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Datasets.
Revised: Datasets.

Original: We conduct experiments on multiple datasets for image classiﬁcation and text classiﬁcation tasks.
Revised: We conduct experiments on multiple datasets for image classiﬁcation and text classiﬁcation tasks.

Original: Following Geifman & El-Yaniv (2017; 2019)
Revised: Following Geifman & El-Yaniv (2017; 2019)

Original: ; Liu et al.
Revised: ; Liu et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2020), we use CIFAR-10, CIFAR-100 (Krizhevsky, 2009), and SVHN (Netzer et al., 2011) for image classiﬁcation tasks, and following Xin et al.
Revised: (2020), we use CIFAR-10, CIFAR-100 (Krizhevsky, 2009), and SVHN (Netzer et al., 2011) for image classiﬁcation tasks, and following Xin et al.

Original: (2021), we use MRPC (Dolan & Brockett, 2005), MNLI (Williams et al., 2018), and QNLI (Wang et al., 2018) for text classiﬁcation tasks.
Revised: (2021), we use MRPC (Dolan & Brockett, 2005), MNLI (Williams et al., 2018), and QNLI (Wang et al., 2018) for text classiﬁcation tasks.

Original: The usage of the training set and test set in selective classiﬁcation are the same as the standard classiﬁcation, because current selective classiﬁcation focuses on in-domain data (i.e., data from the same distribution as the training set)
Revised: The usage of the training set and test set in selective classiﬁcation are the same as the standard classiﬁcation, because current selective classiﬁcation focuses on in-domain data (i.e., data from the same distribution as the training set)

Original: (Geifman & El-Yaniv, 2017).
Revised: (Geifman & El-Yaniv, 2017).

Original: For example, if the selective classiﬁer is trained on the training set of CIFAR-10 (Krizhevsky, 2009), then it will be tested on the test set of CIFAR-10.
Revised: For example, if the selective classiﬁer is trained on the training set of CIFAR-10 (Krizhevsky, 2009), then it will be tested on the test set of CIFAR-10.

Original: Furthermore, MNLI’s development set and test set are divided into matched and mismatched parts.
Revised: Furthermore, MNLI’s development set and test set are divided into matched and mismatched parts.

Original: The matched parts are sampled from the same source as the training set (so they are in-domain data), while the mismatched parts are sampled from different sources.
Revised: The matched parts are sampled from the same source as the training set (so they are in-domain data), while the mismatched parts are sampled from different sources.

Original: In our experiments, only the matched parts are usedin experiments, since the current selective classiﬁcation only considers indomain data.
Revised: In our experiments, only the matched parts are used since the current selective classiﬁcation only considers in-domain data.

Original: In addition, test sets of MRPC, QNLI, and MNLI are not accessible, so we use their development sets as test sets.
Revised: In addition, test sets of MRPC, QNLI, and MNLI are not accessible, so we use their development sets as test sets.

Original: Following Liu et al.
Revised: Following Liu et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2020), since CIFAR-10, 
Revised: (2020), since CIFAR-10, CIFAR-100 and

Original: CIFAR-100 and SVHN originally had no development set, their development sets were 2000 samples randomly split from corresponding test sets.
Revised: SVHN originally had no development set, their development sets were 2000 samples randomly split from corresponding test sets.

Original: More details of all the datasets in our experiments are described in Appendix C.1.
Revised: More details of all the datasets in our experiments are described in Appendix C.1.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: The evaluation metrics are AURC and selective risk (the lower the better for both).
Revised: The evaluation metrics are AURC and selective risk (the lower, the better for both).

Original: AURC is a comprehensive metric of selective classiﬁer, and selective risk is a common metric in previous works (Geifman & El-Yaniv, 2019; Liu et al., 2019; Huang et al., 2020).
Revised: AURC is a comprehensive metric of selective classiﬁers, and selective risk is a standard metric in previous works (Geifman & El-Yaniv, 2019; Liu et al., 2019; Huang et al., 2020).

Original: In this paper, given a selective classiﬁer, the result of selective risk is shown in the form of risk-coverage curves, which shows the selective risk of the selective classiﬁer against its coverage.
Revised: In this paper, given a selective classiﬁer, the result of selective risk is shown in the form of risk-coverage curves, which shows the selective risk of the selective classiﬁer against its coverage.

Original: According to the object of selective classiﬁcation, a selective classiﬁer with a lower risk-coverage curve is better.
Revised: According to the object of selective classiﬁcation, a selective classiﬁer with a lower risk-coverage curve is better.

Original: Networks.
Revised: Networks.

Original: Following Huang et al.
Revised: Following Huang et al.

Original: ; Xin et al.
Revised: ; Xin et al.

Original: (2021), for image classiﬁcation and text classiﬁcation, we use VGG-16 (Simonyan & Zisserman, 2014) and BERT-base (Devlin et al., 2019) as  backbones of selective classiﬁers respectively.
Revised: (2021), for image classiﬁcation and text classiﬁcation, we use VGG-16 (Simonyan & Zisserman, 2014) and BERT-base (Devlin et al., 2019) as the backbones of selective classiﬁers, respectively.

Original: More details of the backbone models and their training procedures are provided in Appendix C.2.
Revised: More details of the backbone models and their training procedures are provided in Appendix C.2.

Original: Baselines.
Revised: Baselines.

Original: We use SR (Geifman & El-Yaniv, 2017), SN (SelectiveNet) (Geifman & El-Yaniv, 2019),
Revised: We use SR (Geifman & El-Yaniv, 2017), SN (SelectiveNet) (Geifman & El-Yaniv, 2019),

Original: Gambler (Liu et al., 2019), SAT (Huang et al., 2020), and Reg-curr (Xin et al., 2021) for both image classiﬁcation and text classiﬁcation.
Revised: Gambler (Liu et al., 2019), SAT (Huang et al., 2020), and Reg-curr (Xin et al., 2021) for both image classiﬁcation and text classiﬁcation.

Original: Note that the SN is optimized fora ﬁxed coverage, so the comprehensive metrics AURC and risk-coverage curve, which summarizes performances under different coverages, are not suitable for evaluating the SN.
Revised: Note that the SN is optimized for ﬁxed coverage, so the comprehensive metrics AURC and risk-coverage curve, which summarizes performances under different coverages, are not suitable for evaluating the SN.

Original: Thus, we only evaluate the selective risk for a ﬁxed coverage of the SN ensemble and provide the results in Appendix D. To ensure a fair comparison, for tasks of image classiﬁcation, all the baselines are re-implemented based on the open resource code of SAT (Huang et al., 2020), and for tasks of text classiﬁcation, they are re-implemented based on the open resource code of Reg-curr (Xin et al., 2021).
Revised: Thus, we only evaluate the selective risk for a ﬁxed coverage of the SN ensemble and provide the results in Appendix E. To ensure a fair comparison, for tasks of image classiﬁcation, all the baselines are re-implemented based on the open resource code of SAT (Huang et al., 2020), and for tasks of text classiﬁcation, they are re-implemented based on the open resource code of Reg-curr (Xin et al., 2021).

Original: The details of  hyperparameters of each baseline are provided in Appendix C.3.
Revised: The details of the hyperparameters of each baseline are provided in Appendix C.3.

Original: 6. V ERIFICATION OF THE A SSUMPTIONS
Revised: 6. V ERIFICATION OF THE A SSUMPTIONS

Original: We examine Assumption 1 and Assumption 2 ( only for the baseline of SR) on datasets for both image classiﬁcation and text classiﬁcation.
Revised: We examine Assumption 1 and Assumption 2 ( only for the baseline of SR) on datasets for both image classiﬁcation and text classiﬁcation.

Original: Since the deﬁnite samples are the idealization of samples on which member models provide almost the same predictive probability distributions, we take samples with a standard deviation of predictive probability distributions of member models (or STD for short) less than a small positive number (cid:15) as deﬁnite samples and the other samples as ambiguous samples in experiments.
Revised: Since the deﬁnite samples are the idealization of samples on which member models provide almost the same predictive probability distributions, we take samples with a standard deviation of predictive probability distributions of member models (or STD for short) less than a small positive number (cid:15) as deﬁnite samples and the other samples as ambiguous samples in experiments.

Original: Formally, STD := (cid:113)
Revised: Formally, STD := (cid:113)

Original: [Equation]
Revised: [Equation]

Original: , where ˆ π j is the predictive probability distribution vector of the j -th member model, and samples with STD < (cid:15) approximates deﬁnite samples in experiments.
Revised: , where ˆ π j is the predictive probability distribution vector of the j -th member model, and samples with STD < (cid:15) approximates deﬁnite samples in experiments.

Original: We choose (cid:15) =
Revised: We choose (cid:15) =

Original: 10 − 3 for datasets of image classiﬁcation and (cid:15) =
Revised: 10 − 3 for datasets of image classiﬁcation and (cid:15) =

Original: 10 − 2 for datasets of text classiﬁcation.
Revised: 10 − 2 for datasets of text classiﬁcation.

Original: Figure 1(a) and Figure 1(e) show the selective error rates (selective risks) of samples with STD < 10 − 3 and samples with STD ≥ 10 − 3 given a range of conﬁdence thresholds (which approximate Pr( Err | D, C ≥ τ ) and Pr( Err | A, C ≥ τ ) ) on the test set of each dataset.
Revised: Figure 1(a) and Figure 1(e) show the selective error rates (selective risks) of samples with STD < 10 − 3 and samples with STD ≥ 10 − 3 given a range of conﬁdence thresholds (which approximate Pr( Err | D, C ≥ τ ) and Pr( Err | A, C ≥ τ ) ) on the test set of each dataset.

Original: The results show that the selective risk of samples with STD < 10 − 3 is lower than that of samples with STD ≥ 10 − 3 for all conﬁdence thresholds near 1 on all datasets except
Revised: The results show that the selective risk of samples with STD < 10 − 3 is lower than that of samples with STD ≥ 10 − 3 for all conﬁdence thresholds near 1 on all datasets, which veriﬁes

Original: MRPC, which veriﬁes Assumption 1.
Revised: Assumption 1.

Original: Figure 1(b)-1(d) and Figure 1(f)-1(h) shows the histogram of conﬁdence scores of samples with STD < (cid:15) and that of other samples, which approximate p C ( κ | D ) and p C ( κ | A ) , on the test set of each dataset.
Revised: Figure 1(b)-1(d) and Figure 1(f)-1(h) shows the histogram of conﬁdence scores of samples with STD < (cid:15) and that of other samples, which approximate p C ( κ | D ) and p C ( κ | A ) , on the test set of each dataset.

Original: The results show that the number of samples with STD < 10 − 3 is non-zero in the top bin on all datasets except MRPC, which veriﬁes Assumption 2.
Revised: The results show that the number of samples with STD < 10 − 3 is non-zero in the top bin on all datasets, which veriﬁes Assumption 2.

Original: In summary, Assumption 1 and 2 holds on all datasets except MRPC.
Revised: In summary, Assumption 1 and 2 hold on all datasets. [Figure]

Original: 6. E VALUATION OF S ELECTIVE C LASSIFIER E NSEMBLES
Revised: 6. E VALUATION OF S ELECTIVE C LASSIFIER E NSEMBLES

Original: Table 1 shows the AURCs of the individual model and the ensembles of 2-5 member models for each baseline on each dataset, where o is the hyperparameter of Deep Gambler.
Revised: We ﬁrst verify Theorem 2. Figure 2 shows the risk-coverage curves of the ensembles and the individual models of each baseline on each dataset.

Original: The results show that the AURC of the ensemble is signiﬁcantly lower than that of the individual  model in each case except the case of Gambler on MRPC.
Revised: As we can see, except on MRPC, the risk-coverage curve of the ensemble is always entirely below that of the individual model, i.e., the ensemble has a lower selective risk than the individual model under any coverage, which is consistent with Theorem 2. The abnormal results on MRPC dataset may be because of the small number of samples in MRPC.

Original: On CIFAR-10, CIFAR-100, SVHN, MRPC, QNLI, and MNLI, the 
Revised: The development set of MRPC has only 0.4k samples, which is much smaller than development sets of other datasets (see Table 2). More importantly, when the coverage is low, say 10%, only about 40 samples in MRPC are selected to predict, which may cause a large variance in selective risk estimation. Thus, the estimation of selective risk is not accurate under low coverage, which may explain the violation of Theorem 2 on MRPC. In summary, except the results on MRPC, which might have a large variance in selective risk estimation, the experimental results in Figureverify the correctness and practicability of Theorem 2.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: AURCs of the best ensemble models of 5 members are respectively 23%, 15%, 7%, 11%, 14%
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: and 9% lower than those of the best individual models. Among all ensembles, SR ensemble has the lowest AURCs on CIFAR-10 and CIFAR-100, SAT ensemble has the lowest AURCs on SVHN 1 , and Reg-curr has the lowest AURCs on all datasets for text classiﬁcation.
Revised: [Table]

Original: Risk-coverage curves of the ensembles and the individual models of each baseline on each dataset are shown in Figure 2, which shows that for each baseline on each dataset except MRPC, the risk-coverage curve of the ensemble is almost completely below that of the individual model.
Revised: [Figure] [Table] [Figure] [Table]

Original: In other words, on each dataset except MRPC, the ensemble has a lower selective risk than the individual model under any coverage.
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: The experimental results for SR in Figure 2 verify the correctness and practicability of Theorem 2. The assumptions hold on each dataset except MRPC. Combining this with Theorem 2, the SR ensemble should have a lower selective risk than the individual model under a range of coverage (from 0 to some φ 0 ) on each dataset except MRPC. This is supported by the experimental results
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: Next, we show the AURCs of the ensembles in Table 1, where o is the hyperparameter of Deep

Revised: Gambler, and each ensemble has 2-5 member models.

Revised: As we can see, the ensemble with ﬁve members has a signiﬁcantly lower AURC than the individual model on each dataset.

Original: It is surprising that the SR ensemble and Reg-curr ensemble are state-of-the-art selective classiﬁers for image and text classiﬁcation tasks respectively.
Revised: On CIFAR-10, CIFAR-100, SVHN, MRPC, QNLI, and MNLI, the AURCs of the best ensemble models with ﬁve members are 23%, 15%, 7%, 14%, 14%, and 9% lower than those of the best individual models. Among all ensembles, SR ensemble has the lowest AURCs on CIFAR-10, CIFAR-100, and MRPC, Surprisingly, the SR ensemble and Reg-curr ensemble are state-of-the-art selective classiﬁers for image and text classiﬁcation tasks, respectively.

Original: They only use the maximum probability as the conﬁdence score, while SAT and SN use more sophisticated and explicit conﬁdence score function.
Revised: They only use the maximum probability as the conﬁdence score, while SAT and SN use more sophisticated and explicit conﬁdence score functions.

Original: Note that, on SVHN, the SAT ensemble performs better than the SR ensemble.
Revised: Note that, on SVHN, the SAT ensemble performs better than the SR ensemble.

Original: We ﬁnd that the annotations on SVHN are noisy, and after manually removing some noisy samples in the training set,  SR ensemble performs better than SAT ensemble.
Revised: We ﬁnd that the annotations on SVHN are noisy, and after manually removing some noisy samples in the training set, the SR ensemble performs better than SAT ensemble.

Original: Therefore, the SVHN result indicates that
Revised: Therefore, the SVHN result indicates that

Original: SAT is better at handling label noise rather than selecting when to abstain.
Revised: SAT is better at handling label noise rather than selecting when to abstain.

Original: Please refer to Appendix
Revised: Please refer to Appendix

Original: E.3 for more details.
Revised: F.3 for more details.

Original: In addition, we conduct experiments to explore further properties of the selective classiﬁer ensemble, including the effect of the number of members and the relationship between the classiﬁcation performance and the selective classiﬁcation performance (see Appendix E).
Revised: In addition, we conduct experiments to explore further properties of the selective classiﬁer ensemble, including the effect of the number of members and the relationship between the classiﬁcation performance and the selective classiﬁcation performance (see Appendix F).

Original: The results are that an ensemble with more member models hasa better selective performance, and good classiﬁcation performance of the ensemble does not necessarily imply good selective classiﬁcation performance.
Revised: The results are that an ensemble with more member models has better selective performance, and good classiﬁcation performance of the ensemble does not necessarily imply good selective classiﬁcation performance.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: The experimental results suggest that the assumptions may capture the essential reason for the better performance of the ensemble than the individual selective classiﬁer.

Original: According to the analysis result, the assumptions are a sufﬁcient condition for good performance of the ensemble, but no analysis shows that they are a necessary condition.

Original: However, the experiments show that the assumptions might be a necessary condition, since the ensemble only does not perform well on MRPC, the only dataset where the assumptions do not hold (see Figure 2 and Figure 1). Thus, the assumptions might capture the essential reason why the ensemble works. Based on this work, there are two possible directions for future work. First, the theoretical result in this paper may motivate the analysis of the randomization-based ensemble for standard classiﬁcationin practical settings .
Revised: A possible direction for future work is to adapt our analysis to standard classiﬁcation .

Original: The previous analyses of the randomization-based ensemble for standard classiﬁcation need some impractical assumptions (Tumer & Ghosh, 1996; Fumera & Roli, 2005).
Revised: The previous analyses of the randomization-based ensemble for standard classiﬁcation need some impractical assumptions (Tumer & Ghosh, 1996; Fumera & Roli, 2005).

Original: On the contrary,the assumptions in this paper are a good approximation of practical settings (see Section 6.1), and more importantly, the standard classiﬁcation is a special case of selective classiﬁcation, i.e. selective classiﬁcation with coverage of 1.
Revised: On the contrary, this paper’s assumptions are a good approximation of practical settings (see Section 6.1), and more importantly, the standard classiﬁcation is a particular case of selective classiﬁcation, i.e., selective classiﬁcation with coverage of 1.

Original: Therefore, our analysis (although it does not cover the case of coverage of 1, i.e. the case of standard classiﬁcation) may motivate the analysis of the randomization-based ensemble for standard classiﬁcation in practical settings. 
Revised: Therefore, our analysis (although it does not cover the case of coverage of 1, i.e., the case of standard classiﬁcation) may motivate the analysis of the randomization-based ensemble for standard classiﬁcation in practical settings. Another possible direction for future work is the relaxation of assumptions.

Original: Second, it is interesting to relax the assumption of the idealization of the deﬁnitesamples and then show a similar theoretical result. The idealization is a strong assumption that all member models provide the same predictive probability distribution on deﬁnite samples.
Revised: This paper’s assumption is a little strong for the convenience of proof. Although the experimental results suggest that these assumptions are the actual behaviors of the SR model, we guess the assumptions can be relaxed while the conclusion keeps the same. It is interesting to relax the assumption of the idealization of the deﬁnite samples.

Original: Although it may be a good approximation of practical setting, it is unrealistic anyway.
Revised: Although the idealization may be a good approximation of practical setting, it is unrealistic anyway.

Original: However, we believe that a similar theoretical result holds in the absence of the idealization, given a set of assumptions similar to the assumptions in this paper.
Revised: We believe that a similar theoretical result holds in the absence of the idealization.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We prove that under some assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.
Revised: We prove that under some assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.

Original: Although the metrics of selective classiﬁcation are non-convex, we complete the proof with the help of several assumptions motivated by empirical observations.
Revised: Although the metrics of selective classiﬁcation are non-convex, we complete the proof with the help of several assumptions motivated by empirical observations.

Original: The assumptions and the result are well supported by the experimental results on multiple datasets of image classiﬁcation tasks and text classiﬁcation tasks.
Revised: The assumptions and the result are well supported by the experimental results on multiple datasets of image classiﬁcation tasks and text classiﬁcation tasks.

Original: A surprising empirical result is that two simple methods, SR ensemble and its variant Reg-curr ensemble, (which can be summarized as the ensemble models with maximum probability as conﬁdence ) are state-of-the-art selective classiﬁers
Revised: A surprising empirical result is that two simple methods, SR ensemble and its variant Reg-curr ensemble, (which can be summarized as the ensemble models with maximum probability as conﬁdence ) are state-of-the-art selective classiﬁers.

Original: A R ELATIONSHIP BETWEEN SELECTIVE CLASSIFICATION AND OOD DETECTION
Revised: A R ELATIONSHIP BETWEEN SELECTIVE CLASSIFICATION AND OOD DETECTION

Original: Figure 3 shows the relationship between selective classiﬁcation and OOD detection (or open set recognition, novelty detection).
Revised: Figure 3 shows the relationship between selective classiﬁcation and OOD detection (or open set recognition, novelty detection).

Original: B P ROOFS
Revised: B P ROOFS

Revised: The complete proof is somewhat complex but its intuition is straightforward.

Revised: In a word, the intuition is that because the ensemble avoids to be overconﬁdent over ambiguous samples, the ensemble has a lower selective risk.

Original: We prove Proposition 1 in Section B.1 and  prove Theorem 2 in Section B.2.
Revised: Details of the intuition are as follows: the individual model is not ”modest” over both ambiguous samples and deﬁnite samples (Assumption 2); by contrast, based on Assumption 3, we prove that the ensemble provides modest conﬁdence to ambiguous samples (Proposition 1). In addition, the conﬁdence of deﬁnite samples remains the same throughout ensembling (due to the deﬁnition of deﬁnite samples); 3.

Revised: thus, when conﬁdence approaches 1, as long as the the classiﬁer’s error rate over deﬁnite samples is lower than the error rate over ambiguous samples (Assumption 1), the individual model suffers more error predictions that come with the ambiguous samples than the ensemble.

Revised: Based on this, we prove that the selective risk drops under a range of coverage via ensembling (Theorem 2).

Revised: We prove Proposition 1 in Section B.1 and prove Theorem 2 in Section B.2.

Original: The road map of the complete proof of the ﬁnal result, i.e. Theorem 2, is shown in Figure 4.
Revised: The road map of the complete proof of the ﬁnal result, i.e., Theorem 2, is shown in Figure 4.

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: To prove Proposition 1,  we  show that for ambiguous samples, the ensemble is conservative in providing the predictive probability of each class at ﬁrst.
Revised: To prove Proposition 1, that is, the ensemble is modest over ambiguous samples, we ﬁrst show that over ambiguous samples, the ensemble provides a moderate predictive probability for each class.

Original: By ’conservative’, we mean that the predictive probability provided by the model is modest, i.e. away from the extreme values 0 and 1.
Revised: By moderate predictive probability , we mean a predictive probability whose PDF approaches zero when itself approaches one or zero 4 .

Original: Formally, we have the following lemma (proof is provided in Section B.1.1).
Revised: Formally, we have the following lemma (proof is provided in Section B.1.1).

Original: Lemma 3.
Revised: Lemma 3.

Original: If Assumption 3 holds, then
Revised: If Assumption 3 holds, then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (11) and
Revised: (11) and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  where the notation follows that of Assumption 3, and Π k ens is the predictive probability of the ensemble for class k .
Revised: (13) where the notation follows that of Assumption 3, and Π k ens is the predictive probability of the ensemble for class k .

Original: Secondly,  we show the relation of the PDF ofthe conﬁdence to those of predictive probabilities.
Revised: Secondly, to prove the ensemble is modest over ambiguous samples, we show the relationship between the PDF of conﬁdence and the PDFs of predictive probabilities.

Original: Note that for an SR model, its conﬁdence is the maximumof predictive probabilities of all classes.
Revised: Note that the conﬁdence of an SR model is the maximum predictive probabilities.

Original: With the following lemma (proof is provided in Section B.1.2), we can bound the PDF ofthe conﬁdence by those of predictive probabilities.
Revised: Thus, the following lemma (proof is provided in Section B.1.2) bounds the PDF of conﬁdence by PDFs of predictive probabilities, and then it is clear that the ensemble is modest over ambiguous samples, considering the moderate predictive probabilities of the ensemble over ambiguous samples.

Original: Lemma 4.
Revised: Lemma 4.

Original: Let Π k (1 ≤ k ≤ K ) be K continuous random variables, and C :=
Revised: Let Π k (1 ≤ k ≤ K ) be K continuous random variables, and C :=

Original: max k Π k .
Revised: max k Π k .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Finally, since C ens = max k Π k ens , Lemma 3 and Lemma 4 derive that when κ ens → 1 − ,
Revised: Finally, since C ens = max k Π k ens , Lemma 3 and Lemma 4 derive that when κ ens → 1 − ,

Original: [Equation]
Revised: [Equation]

Original: and thus Proposition 1 holds.
Revised: and thus Proposition 1 holds.

Original: B.1. P ROOF OF L EMMA
Revised: B.1. P ROOF OF L EMMA

Original: We ﬁrst derive the PDF of the average of multiple continuous random variables in terms of the PDFs of these random variables.
Revised: We ﬁrst derive the PDF of the average of multiple continuous random variables in terms of the PDFs of these random variables (Lemma 5), which helps us to analyze the PDF of the ensemble’s predictive probabilities.

Original: Lemma 5.
Revised: Lemma 5.

Original: Let X 1 , X 2 , . . .
Revised: Let X 1 , X 2 , . . .

Original: X M be M continuous random variables, and their average is X avg := 1 M (cid:80) Mi
Revised: X M be M continuous random variables, and their average is X avg := 1 M (cid:80) Mi

Original: =1 X i .
Revised: =1 X i .

Original: Then the PDF of X avg is
Revised: Then the PDF of X avg is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where p (cid:126) X is p X 1 , X 2 ,..., X M for short.
Revised: where p (cid:126) X is p X 1 , X 2 ,..., X M for short.

Original: Proof.
Revised: Proof.

Original: The distribution function of X avg is
Revised: The distribution function of X avg is

Original: [Equation]
Revised: [Equation]

Original: Let x M = u − (cid:80) M − 1 i =1 x i , then the integral above is equal to
Revised: Let x M = u − (cid:80) M − 1 i =1 x i , then the integral above is equal to

Original: [Equation]
Revised: [Equation]

Original: The PDF of X avg is the derivative of F X avg , which combined with (16) derives
Revised: The PDF of X avg is the derivative of F X avg , which, combined with (,16) derives

Original: [Equation]
Revised: [Equation]

Original: which is exactly (15).
Revised: which is exactly (15) [Equation]

Original:  Based on Lemma 5, we prove Lemma 3 as follows.
Revised: Proof of Lemma 3. Based on Lemma 5 and Assumption 3, we prove Lemma 3 as follows.

Original: Proof.
Revised: Proof.

Original: With Lemma 5 applied to Π ki , 1 ≤ i ≤ M , we have
Revised: With Lemma 5 applied to Π ki , 1 ≤ i ≤ M , we have

Original: [Equation]
Revised: [Equation]

Original: where p (cid:126) Π k is p Π k 1 ,..., Π kM for short.
Revised: where p (cid:126) Π k is p Π k 1 ,..., Π kM for short.

Original: The integrand in the right-hand side of (17) is non-zero requires
Revised: The integrand in the right-hand side of (17) being non-zero requires

Original: [Equation]
Revised: [Equation]

Original: Firstly [Equation] we prove (10) and (12).
Revised: Firstly, we prove (10) and (12).

Original: When 0 ≤ π k ens ≤ 1 M , it easy to verify that (18) is equivalent to
Revised: When 0 ≤ π k ens ≤ 1 M , it easy to verify that (18) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Thus, (17) transforms into 
Revised: Thus, (17) transforms into p

Original: [Equation]
Revised: [Equation]

Original: Considering that p (cid:126) Π k ( ·| A ) is bounded, as Assumption 3 claims, let B be one of its upper bounds.
Revised: Considering that p (cid:126) Π k ( ·| A ) is bounded, as Assumption 3 claims, let B be one of its upper bounds.

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: which directly derives (10) and (12) (note that the PDF is non-negative).
Revised: which directly derives (10) and (12) (note that the PDF is non-negative).

Original: Secondly, we prove (11) and (13).
Revised: Secondly, we prove (11) and (13).

Original: These two equations could be derived like (10) and (12).
Revised: These two equations could be derived like (10) and (12).

Original: However, here we take another way that uses a little trick to simplify the proof.
Revised: However, here we take another way that uses a little trick to simplify the proof.

Original: Let Π ki and Π k ens be the corresponding random variables of π ki and π k ens respectively, and U i =
Revised: Let Π ki and Π k ens be the corresponding random variables of π ki and π k ens respectively, and U i =

Original: 1 − Π ki ,
Revised: 1 − Π ki ,

Original: U ens = 1 M (cid:80) Mi
Revised: U ens = 1 M (cid:80) M i

Original: =1 U i =
Revised: [Equation] U i =

Original: 1 − Π k ens .
Revised: 1 − Π k ens .

Original: Applying (20) to U i and U ens , we get that when 0 ≤ u ens ≤ 1 M ,
Revised: Applying (20) to U i and U ens , we get that when 0 ≤ u ens ≤ 1 M ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: It is easy to see that p U ens ( u ens | A ) =
Revised: It is easy to see that p U ens ( u ens | A ) =

Original: p Π k ens (1 − u ens | A ) .
Revised: p Π k ens (1 − u ens | A ) .

Original: Sticking this into (21), we have
Revised: Sticking this into (21), we have

Original: [Equation]
Revised: [Equation]

Original: when 0 ≤ u ens ≤ 1 M .
Revised: when 0 ≤ u ens ≤ 1 M .

Original: With the (1 − u ens ) in the equation above replaced with π k ens , we have
Revised: With the (1 − u ens ) in the equation above replaced with π k ens , we have

Original: [Equation]
Revised: [Equation]

Original: when 1 − 1 M ≤ π k ens ≤ 1 , which directly derives (11) and (13).
Revised: when 1 − 1 M ≤ π k ens ≤ 1 , which directly derives (11) and (13).

Original: B.1. P ROOF OF L EMMA
Revised: B.1. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: First of all, we prove ∀ κ 1 , 
Revised: First of all, we prove ∀ κ 1 , κ 2 , κ 1 < κ 2 ,

Original: [Equation]
Revised: [Equation]

Original: It is easy to see that
Revised: It is easy to see that

Original: [Equation]
Revised: [Equation]

Original: so the left-hand side of (22) is 
Revised: so the left-hand side of (22) is (cid:90)

Original: [Equation]
Revised: [Equation]

Original: where the last equality is due to ( −∞ , κ 1 ] ⊂ ( −∞ , κ 2 ] [Equation] and the right-hand side of (22) is 
Revised: where the last equality is due to ( −∞ , κ 1 ] ⊂ ( −∞ , κ 2 ] , and the right-hand side of (22) is K (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because R k − 1 × [ κ 1 , κ 2 ] × R K − k for different k , 1 ≤ k ≤ K , may have an intersection.
Revised: where the last inequality is because R k − 1 × [ κ 1 , κ 2 ] × R K − k for different k , 1 ≤ k ≤ K , may have an intersection.

Original: To prove (22), we only need to prove that the right-hand side of (23) is less than or equal to the right-hand side of (24), which is equivalent to prove 
Revised: To prove (22), we only need to prove that the right-hand side of (23) is less than or equal to the right-hand side of (24), which is equivalent to prove [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now we prove (25).
Revised: Now we prove (25).

Original: ∀ ( π 1 , . . .
Revised: ∀ ( π 1 , . . .

Original: , π K ) ∈ ( −∞ , κ 2 ] K \ ( −∞ , κ 1 ] K , we have 
Revised: , π K ) ∈ ( −∞ , κ 2 ] K \ ( −∞ , κ 1 ] K , we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: 27) where (27) is because if all π k is less than or equal to κ 1 instead, then ( π 1 , . .
Revised: 27) where (27) is because if all π k is less than or equal to κ 1 instead, then ( π 1 , . .

Original: . , π K ) ∈ ( −∞ , κ 1 ] K , which contradicts with ( π 1 , . .
Revised: . , π K ) ∈ ( −∞ , κ 1 ] K , which contradicts with ( π 1 , . .

Original: . , π K ) ∈ ( −∞ , κ 2 ] K \ ( −∞ , κ 1 ] K .
Revised: . , π K ) ∈ ( −∞ , κ 2 ] K \ ( −∞ , κ 1 ] K .

Original: Thus, π k 0 ∈ [ κ 1 , κ 2 ] , so 
Revised: Thus, π k 0 ∈ [ κ 1 , κ 2 ] , so ( π 1 , . . .

Original: [Equation]
Revised: , π K ) ∈ R k 0 − 1 × [ κ 1 , κ 2 ] × R K − k 0 ⊂ K (cid:91) k =

Original:  , which is exactly (25), and therefore (22) is proved.
Revised: R k − 1 × [ κ 1 , κ 2 ] × R K − k , which is precisely (25), and therefore (22) is proved.

Original: With (22) and the deﬁnition of derivatives, it is easy to see that F (cid:48) C ( κ ) ≤ (cid:80) Kk =1 F (cid:48)
Revised: With (22) and the deﬁnition of derivatives, it is easy to see that F (cid:48) C ( κ ) ≤ (cid:80) K k =1 F (cid:48)

Original: Π k ( κ ) , which is equivalent to p C ( κ ) ≤ (cid:80) [Equation]
Revised: Π k ( κ ) , which is equivalent to p C ( κ ) ≤ (cid:80) Kk =1 p Π k ( κ ) .

Original: Thus, Lemma 4 is proved.
Revised: Thus, Lemma 4 is proved.

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Revised: The intuition of the proof is as follows.

Revised: Intuitively, since the ensemble is more modest than the individual over ambiguous samples and is the same modest as the individual model over deﬁnite samples, the ensemble tends to select more deﬁnite samples when the conﬁdence threshold approaches 1, compared with the individual model (Lemma 7).

Revised: Thus, (still intuitively) as long as the selective risk over deﬁnite samples is lower than that over ambiguous samples when the conﬁdence threshold approaches 1 (Assumption 1), the ensemble is certainly to have a lower selective risk when the conﬁdence threshold approaches 1.

Revised: Although the intuition is straightforward, the rigorous proof is not easy.

Original: To prove Theorem 2, we show the following lemmas ﬁrst.
Revised: For the convenience of the proof, we show Lemma 6 and Lemma 7 ﬁrst.

Revised: Lemma 6 claims that for the individual model, the selective risk given deﬁnite samples is lower than the overall selective risk.

Revised: Lemma 7 claims that the ensemble is unlikely to select ambiguous samples to predict when the conﬁdence threshold approaches 1.

Original: Lemma 6.
Revised: Lemma 6.

Original: If Assumption 1-2 holds, then for any individual SR model,
Revised: If Assumption 1-2 holds, then for any individual SR model,

Original: [Equation]
Revised: [Equation]

Original: where the notation follows those of Assumption 1-2.
Revised: where the notation follows those of Assumption 1-2.

Original: Proof.
Revised: Proof.

Original: Using Bayes’ rule, we have
Revised: Using Bayes’ rule [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Using L’Hospital’s rule
Revised: Using L’Hospital’s rule

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to Assumption 2.
Revised: where the last inequality is due to Assumption 2.

Original: Combining this with (29)
Revised: Combining this with (29)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now we derive (28).
Revised: Now we derive (28).

Original: [Equation]
Revised: [Equation]

Original: According to the equation above, we have
Revised: According to the equation above, we have

Original: [Equation]
Revised: [Equation]

Original: Due to (31) and Assumption 1, the ﬁrst term of the equation above is positive, so (28) is derived.
Revised: Due to (31) and Assumption 1, the ﬁrst term of the equation above is positive, so (28) is derived.

Original: Lemma 7.
Revised: Lemma 7.

Original: If Assumption 2-3 hold, then
Revised: If Assumption 2-3 hold, then

Original: [Equation]
Revised: [Equation]

Original: where the notation follows those of Assumption 2-3.
Revised: where the notation follows those of Assumption 2-3.

Original: Proof.
Revised: Proof.

Original: Similar to (31), with Bayes’ rule and L’Hospital’s rule, we can derive that
Revised: Similar to (31), with Bayes’ rule and L’Hospital’s rule, we can derive that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where C ens is the conﬁdence score of the ensemble.
Revised: where C ens is the conﬁdence score of the ensemble.

Original: Because for a deﬁnite sample, the conﬁdence score of the ensemble is equal to that of the individual model, we have
Revised: Because for a deﬁnite sample, the conﬁdence score of the ensemble is equal to that of the individual model, we have

Original: [Equation]
Revised: [Equation]

Original: where the last equality is due to Proposition 1 and Assumption 2.
Revised: where the last equality is due to Proposition 1 and Assumption 2.

Original: Substituting this to (34), we obtain (33).
Revised: Substituting this to (34), we obtain (33).

Original: Now we provide the proof ofthe Theorem 2.
Revised: Proof of Theorem 2.

Original: Proof.
Revised: Proof.

Original: First, for the convenience of the proof, given an SR model ( f, g ) , we deﬁne a threshold-tocoverage function ρ ( f,g ) of ( f, g ) that maps the conﬁdence threshold to the corresponding coverage,
Revised: First, for the convenience of the proof, given an SR model ( f, g ) , we deﬁne a threshold-tocoverage function ρ ( f,g ) of ( f, g ) that maps the conﬁdence threshold to the corresponding coverage,

Original: [Equation]
Revised: [Equation]

Original: Second, we prove that ∃ δ ∈ (0 , 1) , ∀ τ ens ∈ (1 − δ, 
Revised: Second, we prove that ∃ δ ∈ (0 , 1) , ∀ τ ens ∈ (1 − δ, 1) ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where C is the conﬁdence score of the individual SR model, Err ind is the event that the individual model makes an error prediction, Err ens is the event that the ensemble makes an error prediction, ρ and ρ ens are the threshold-to-coverage functions of the individual model and the ensemble respectively.
Revised: where C is the conﬁdence score of the individual SR model, Err ind is the event that the individual model makes an error prediction, Err ens is the event that the ensemble makes an error prediction, ρ and ρ ens are the threshold-to-coverage functions of the individual model and the ensemble respectively.

Original: Note that the symbol ρ − 1 denotes the preimage under ρ , rather than the inverse function of ρ .
Revised: Note that the symbol ρ − 1 denotes the preimage under ρ , rather than the inverse function of ρ .

Original: Because when τ ens → 1 − , the coverage of the ensemble  approaches 0, anddue to the coverage of the individual modelwith the conﬁdence threshold of τ is equal to the coverage of the ensemble ( ρ ( τ ) =
Revised: Because when τ ens → 1 − , the coverage of the ensemble ρ ens ( τ ens ) approaches 0, and the coverage of the individual model is equal to the coverage of the ensemble, i.e., ρ ( τ ) =

Original: ρ ens ( τ ens ) ), we have τ → 1 −  .
Revised: ρ ens ( τ ens ) , we have τ → 1 − when τ ens → 1 − .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: In addition, for deﬁnite samples, the conﬁdence score of the ensemble and that of the individual model are the same, and the ensemble and the individual model make error predictions on the same set of samples, i.e. Err ind =
Revised: In addition, for deﬁnite samples, the conﬁdence score of the ensemble and that of the individual model are the same, and the ensemble and the individual model make error predictions on the same set of samples, i.e. Err ind =

Original: Err ens , so
Revised: Err ens , so

Original: [Equation]
Revised: [Equation]

Original: where the second equality is just a variable substitution.
Revised: where the second equality is just a variable substitution.

Original: Finally, we have
Revised: Finally, we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality is due to Lemma 7, the second equality is due to (37) and (38), and the last inequality is due to Lemma 6.
Revised: where the ﬁrst equality is due to Lemma 7, the second equality is due to (37) and (38) [Equation] and the last inequality is due to Lemma 6.

Original: Thus, with (39), it is easy to see that (35) holds.
Revised: Thus, with (39), it is easy to see that (35) holds.

Original: Third, we have
Revised: Third, we have

Original: [Equation]
Revised: [Equation]

Original:  Pr( Err ens | D, C ens ≥
Revised: , D | C ens ≥ τ ens ) = Pr( Err ens | A, C ens ≥

Original: τ ens )
Revised: τ ens )

Original: Pr(  D | C ens ≥ τ ens ) =
Revised: Pr( A | C ens ≥ τ ens ) + Pr( Err ens | D, C ens ≥ τ ens ) Pr( D | C ens ≥ τ ens ) =

Original: Pr( Err ens | A, C ens ≥
Revised: Pr( Err ens | A, C ens ≥

Original: τ ens )
Revised: τ ens )

Original: Pr( A | C ens ≥ τ ens )
Revised: Pr( A | C ens ≥ τ ens )

Original: + Pr( Err ens | D, C ens ≥ τ ens )
Revised: + Pr( Err ens | D, C ens ≥ τ ens )

Original: [1 − Pr( A | C ens ≥ τ ens )] =
Revised: [1 − Pr( A | C ens ≥ τ ens )] =

Original: [Pr( Err ens | A, C ens ≥ τ ens )
Revised: [Pr( Err ens | A, C ens ≥ τ ens )

Original: − Pr
Revised: − Pr

Original: ( Err ens | D, C ens ≥ τ ens )] · Pr( A | C ens ≥ τ ens ) + Pr( Err ens | D, C ens ≥ τ ens )
Revised: ( Err ens | D, C ens ≥ τ ens )] · Pr( A | C ens ≥ τ ens ) + Pr( Err ens | D, C ens ≥ τ ens )

Original: ≤ Pr( A | C ens ≥ τ ens )
Revised: ≤ Pr( A | C ens ≥ τ ens )

Original: + Pr( Err ens | D, C ens ≥ τ ens ) , (40) where the last inequality is because any probability is in [0, 1].
Revised: + Pr( Err ens | D, C ens ≥ τ ens ) , (40) where the last inequality is because any probability is in [0, 1].

Original: Combining this with (35), we have ∃ δ ∈ (0 , 1) , ∀ τ ens ∈ (1 − δ, 1) ,
Revised: Combining this with (35), we have ∃ δ ∈ (0 , 1) , ∀ τ ens ∈ ( [Equation] − δ, 1) ,

Original: [Equation]
Revised: [Equation]

Original:  Pr( Err ens | C ens ≥ τ ens ) is the selective risk of the ensemble given the conﬁdence threshold of τ ens (or given coverage of ρ ens ( τ ens ) ), and Pr( Err ind | C ≥ τ ) is the selective risk of the individual model given the conﬁdence threshold of τ (or given coverage of ρ ( τ )
Revised: where the second inequality is due to (35). Pr( Err ens | C ens ≥ τ ens ) is the selective risk of the ensemble given the conﬁdence threshold of τ ens (or given coverage of ρ ens ( τ ens ) ), and Pr( Err ind | C ≥ τ ) is the selective risk of the individual model given the conﬁdence threshold of τ (or given coverage of ρ ( τ )

Original: =
Revised: =

Original: ρ ens ( τ ens ) ).
Revised: ρ ens ( τ ens ) ).

Original: These two selective risks are under the same coverage
Revised: These two selective risks are under the same coverage

Original: ρ ens ( τ ens ) .
Revised: ρ ens ( τ ens ) .

Original: Thus, (41) is equivalent to that ∃ δ ∈ (0 , 1) , ∀ τ ens ∈ (1 − δ, 1) , the ensemble has a lower selective risk than the individual model given the coverage of φ
Revised: Thus, (41) is equivalent to that ∃ δ ∈ (0 , 1) , ∀ τ ens ∈ (1 − δ, 1) , the ensemble has a lower selective risk than the individual model given the coverage of φ

Original: = ρ ens ( τ ens ) .
Revised: = ρ ens ( τ ens ) .

Original: This  can be simpliﬁed as ∃ δ ∈ (0 , 1) , ∀ φ ∈ (0 ,
Revised: This statement can be simpliﬁed as ∃ δ ∈ (0 , 1) , ∀ φ ∈ (0 ,

Original: ρ ens (1 − δ )) , the ensemble has a lower selective risk than the individual model given the coverage of φ .
Revised: ρ ens (1 − δ )) , the ensemble has a lower selective risk than the individual model, given the coverage of φ .

Original: C D ETAILS OF E XPERIMENTS C. D ATASETS
Revised: C D ETAILS OF E XPERIMENTS C. D ATASETS

Original: The experiments were conducted on multiple data sets of image classiﬁcation and text classiﬁcation.
Revised: The experiments were conducted on multiple data sets of image classiﬁcation and text classiﬁcation.

Original: Thedatasets of image classiﬁcation  are CIFAR-10, CIFAR-100, (Krizhevsky, 2009) and SVHN (Netzer et al., 2011), whose image sizes are all 32 × 32 × 3 pixels.
Revised: The image classiﬁcation datasets are CIFAR-10, CIFAR-100, (Krizhevsky, 2009) and SVHN (Netzer et al., 2011), whose image sizes are all 32 × 32 × 3 pixels.

Original: The datasets of text classiﬁcation are MRPC (Dolan & Brockett, 2005), MNLI (Williams et al., 2018) and QNLI (Wang et al., 2018).
Revised: The datasets of text classiﬁcation are MRPC (Dolan & Brockett, 2005), MNLI (Williams et al., 2018) and QNLI (Wang et al., 2018).

Original: The task of MRPC is to judge whether two paragraphs of text are semantically equivalent or not.
Revised: The task of MRPC is to judge whether two paragraphs of text are semantically equivalent.

Original: MNLI’s task is to judge the inferential relationship of sentences (three categories).
Revised: MNLI’s task is to judge the inferential relationship between sentences (three categories).

Original: The task of QNLI is to determine whether a paragraph has the answer to a given question.
Revised: The task of QNLI is to determine whether a paragraph has the answer to a given question.

Original: The sizes of the training set, development set, and test set of each data set used in experiments are shown in Table 2.
Revised: The sizes of the training set, development set, and test set of each data set used in experiments are shown in Table 2.

Original: MNLI’s development set and test set are divided into matched and mismatched parts.
Revised: MNLI’s development set and test set are divided into matched and mismatched parts.

Original: In the table, (m) represents matched, and (mm) represents mismatched.
Revised: In the table, (m) represents matched, and (mm) represents mismatched.

Original: The matched parts are sampled from the same source as the training set, while the mismatched parts are sampled from different sources.
Revised: The matched parts are sampled from the same source as the training set, while the mismatched parts are sampled from different sources.

Original: Current selective classiﬁcation only considers test samples from the same distribution as the training set, so only the matched parts are used in experiments.
Revised: Current selective classiﬁcation only considers test samples from the same distribution as the training set, so only the matched parts are used in experiments.

Original: In addition, test sets of MRPC, QNLI, and MNLI are not accessible, so we use their development sets as test sets.
Revised: In addition, test sets of MRPC, QNLI, and MNLI are not accessible, so we use their development sets as test sets.

Original: According to Liu et al.
Revised: According to Liu et al.

Original: ; Huang et al.
Revised: Huang et al.

Original: (2020), since CIFAR-10, CIFAR-100 and SVHN originally had no development set, their development sets were 2000 samples randomly divided from corresponding test sets.
Revised: (2020), since CIFAR-10, CIFAR-100 and SVHN originally had no development set, their development sets were 2000 samples randomly divided from corresponding test sets.

Original: [Table]
Revised: [Table]

Original: C
Revised: I MPLEMENTATIONS AND T RAINING P ROCEDURES

Original: M ODEL I MPLEMENTATIONS AND T RAINING P ROCEDURES

Original: For image classiﬁcation, the backbone model is VGG-16 (Simonyan & Zisserman, 2014) with
Revised: For image classiﬁcation, the backbone model is VGG-16 (Simonyan & Zisserman, 2014) with

Original: Dropout (Srivastava et al., 2014), batch normalization (Ioffe & Szegedy, 2015).
Revised: Dropout (Srivastava et al., 2014), batch normalization (Ioffe & Szegedy, 2015).

Original: It is trained in the same way as Huang et al.
Revised: It is trained in the same way as Huang et al.

Original: The model is optimized using SGD with an initial learning rate of 0.1 (the learning rate decays by half in every 25 epochs), the momentum of 0.9, weight decay of 0.0005, batch size of 128, and a total training epoch of 300.
Revised: The model is optimized using SGD with an initial learning rate of 0.1 (the learning rate decays by half in every 25 epochs), the momentum of 0.9, weight decay of 0.0005, batch size of 128, and a total training epoch of 300.

Original: Data preprocessing includes data augmentation (random cropping and ﬂip) and normalization.
Revised: Data preprocessing includes data augmentation (random cropping and ﬂip) and normalization.

Original: The implementations of the backbone model and data preprocessing are based on the ofﬁcial open-sourced implementation of SAT to ensure a fair comparison.
Revised: The implementations of the backbone model and data preprocessing are based on the ofﬁcial open-sourced implementation of SAT to ensure a fair comparison.

Original: For text classiﬁcation, the backbone model of selective classiﬁers is BERT-base (Devlin et al., 2019).
Revised: For text classiﬁcation, the backbone model of selective classiﬁers is BERT-base (Devlin et al., 2019).

Original: Pretrained BERT-base is provided by the Huggingface Transformer Library (Wolf et al., 2020).
Revised: Pretrained BERT-base is provided by the Huggingface Transformer Library (Wolf et al., 2020).

Original: It is trained/ﬁne-tuned in the same way as Xin et al.
Revised: It is trained/ﬁne-tuned in the same way as Xin et al.

Original: The model is trained/ﬁne-tuned using AdamW (Loshchilov & Hutter, 2017) for 3 epochs, with a learning rate of 2 × 10 − 5 , batch size of 32, and the maximum input sequence length of 128.
Revised: (2021), except on dataset MRPC. On QNLI and MNLI, the model is trained/ﬁne-tuned using AdamW (Loshchilov & Hutter, 2017) for 3 epochs, with a learning rate of 2 × 10 − 5 , batch size of 32, and the maximum input sequence length of 128.

Revised: On MRPC, the model is trained/ﬁne-tuned for 10 epoch, with other settings the same as those on

Revised: QNLI and MNLI.

Revised: This unique setting of training epoch is due to the small number of samples in

Revised: MRPC, which makes the training require more epochs to reach convergence on MRPC.

Original: C. H YPERPARAMETERS OF S ELECTIVE C LASSIFIERS
Revised: C. H YPERPARAMETERS OF S ELECTIVE C LASSIFIERS

Original: For the hyperparameter c of SN, we choose c = 0 .
Revised: For the hyperparameter c of SN, we choose c = 0 .

Original: 9 for evaluating its selective risk given the coverage of 90%.
Revised: 9 for evaluating its selective risk, given the coverage of 90%.

Original: The results of SN are reported in Appendix D. For the hyperparameter o of
Revised: The results of SN are reported in Appendix E. For the hyperparameter o of

Original: Gambler, we tune o on validation sets in the same way as Liu et al.
Revised: Gambler, we tune o on validation sets in the same way as Liu et al.

Original: For the hyperparameter α of SAT, we set α = 0 .
Revised: For the hyperparameter α of SAT, we set α = 0 .

Original: 99 , the same as Huang et al.
Revised: 99 , the same as Huang et al.

Original: For the hyperparameter λ of Reg-curr, we set λ = 0 . 05 .
Revised: For the hyperparameter λ of Reg-curr, we set λ = 0 . 05 .

Revised: D A DDITIONAL E XPERIMENTAL R ESULTS

Revised: Table 3 and 4 shows the selective risks of ensembles under coverage 10%-100% on each dataset, where hyperparameters of Gambler are the same as those in Table 1.

Revised: Notably, no ensemble consistently outperforms others under all coverage on all datasets, so it is not easy to tell which ensemble is state-of-the-art in this regard.

Revised: This phenomenon is because different ensembles have similar overall performance but adopt different trade-offs between coverage and selective risk.

Revised: In this case, we need a comprehensive metric, e.g., AURC, to identify the state-of-the-art (see Table 1).

Original: D E MPIRICAL RESULTS OF SN
Revised: E E MPIRICAL RESULTS OF SN

Original: With the hyperparameter c of 0.9, we report the selective risks given the coverage of 90% of SN ensembles and the individual SN in Figure 5.
Revised: With the hyperparameter c of 0.9, we report the selective risks given the coverage of 90% of SN ensembles and the individual SN in Figure 5.

Original: The coverage is set to 90% because the target coverage of the SN is c (Geifman & El-Yaniv, 2019)), and c is 0.9 in our experiments.
Revised: The coverage is set to 90% because the target coverage of the SN is c (Geifman & El-Yaniv, 2019)), and c is 0.9 in our experiments.

Original: The results show that each ensemble of SN have a lower selective risk than the individual SN.[Figure] e [Figure]
Revised: The results show that each ensemble of SN has a lower selective risk than the individual SN.

Original: g e [Figure] E F URTHER P ROPERTIES OF S ELECTIVE C LASSIFIER E NSEMBLE E. T HE E FFECT OF N UMBER OF M EMBERS ON S ELECTIVE C LASSIFIER E NSEMBLE
Revised: F F URTHER P ROPERTIES OF S ELECTIVE C LASSIFIER E NSEMBLE F. T HE E FFECT OF N UMBER OF M EMBERS ON S ELECTIVE C LASSIFIER E NSEMBLE

Original: We evaluate AURCs of  SR ensemble, Gambler ensemble, and SAT ensemble of different numbers of members on CIFAR10, and ﬁnd that an ensemble with more members has  better performance, but is less efﬁcient.
Revised: We evaluate AURCs of the SR ensemble, Gambler ensemble, and SAT ensemble of different numbers of members on CIFAR10, and ﬁnd that an ensemble with more members has a better performance, but is less efﬁcient.

Original: The results are shown in Figure 6.
Revised: The results are shown in Figure 6.

Original: In most cases, the AURC on the test set of CIFAR-10 decreases as the number of members in the ensemble increases.
Revised: In most cases, the AURC on the test set of CIFAR-10 decreases as the number of members in the ensemble increases.

Original: In addition, as the number of members in the ensemble grows, the effect of adding one member drops.
Revised: In addition, as the number of members in the ensemble grows, the effect of adding one member drops.

Original: On the one hand, the result shows that an ensemble with a small number of members has good selective classiﬁcation performance.
Revised: On the one hand, the result shows that an ensemble with a small number of members has good selective classiﬁcation performance.

Original: On the other hand, it indicates that when the number of member models is large, increasing the number of members to improve the performance of the selective classiﬁcation ensemble is not efﬁcient. [Figure]
Revised: On the other hand, it indicates that when the number of member models is large, increasing the number of members to improve the performance of the selective classiﬁcation ensemble is inefﬁcient.

Original: E. G OOD C LASSIFICATION P ERFORMANCE D OES N OT I MPLY G OOD S ELECTIVE C LASSIFICATION P ERFORMANCE
Revised: F. G OOD C LASSIFICATION P ERFORMANCE D OES N OT I MPLY G OOD S ELECTIVE C LASSIFICATION P ERFORMANCE

Original: It is well known that the ensemble has better classiﬁcation performance than an individual model, but this does not guarantee a better selective classiﬁcation performance of the ensemble.
Revised: It is well known that the ensemble has better classiﬁcation performance than an individual model, but this does not guarantee a better selective classiﬁcation performance of the ensemble.

Original: To demonstrate this, we design an SR model with a big backbone, and show that it has as good classiﬁcation  performance as an SR ensemble with a normal backbone but worse selective classiﬁcation performance than an SR model with a normal backbone.
Revised: To demonstrate this, we design an SR model with a big backbone, and show that it has as good classiﬁcation [Table] performance as an SR ensemble with a standard backbone but worse selective classiﬁcation performance than an SR model with a standard backbone.

Original: The big backbone is designed to have twice as many ﬁlters in every convolutional layer and neurons in every fully connected hidden layer as those of the standard VGG-16, which is therefore called Big VGG-16 .
Revised: The big backbone is designed to have twice as many ﬁlters in every convolutional layer and neurons in every fully connected hidden layer as those of the standard VGG-16, which is therefore called Big VGG-16 .

Original: It is easy to see that its number of parameters is approximately 2 2 = 4 times as many as that of standard VGG-16.
Revised: It is easy to see that its number of parameters is approximately 2 2 = 4 times as many as that of standard VGG-16.

Original: We train an
Revised: We train an

Original: SR ensemble of 4 VGG-16s and an SR model with a backbone of Big VGG-16 on CIFAR-10 and show the evaluation results in Figure 7 and Table 3.
Revised: SR ensemble of 4 VGG-16s and an SR model with a backbone of Big VGG-16 on CIFAR-10 and show the evaluation results in Figure 7 and Table 5.

Original: Figure 7 shows that when coverage is high, the ensemble and the big individual model have similar selective risks, and especially, the classiﬁcation error rates (i.e. selective risk of 100% coverage) of the ensemble and the big individual model are similar.
Revised: Figure 7 shows that when coverage is high, the ensemble and the big individual model have similar selective risks, and especially, the classiﬁcation error rates (i.e., selective risk of 100% coverage) of the ensemble and the big individual model are similar.

Original: However, when coverage is low, the big individual model has signiﬁcantly higher selective risk than the ensemble.
Revised: However, when coverage is low, the big individual model has signiﬁcantly higher selective risk than the ensemble.

Original: Table 3 shows that the AURC of Big VGG-16 is much higher than the ensemble of 4 VGG-16s and even higher than SR.
Revised: Table 5 shows that the AURC of Big VGG-16 is much higher than the ensemble of 4 VGG-16s and even higher than SR.

Original: In summary, we show that a selective classiﬁer with a good classiﬁcation performance is not guaranteed to have good selective classiﬁcation performance, so the good selective classiﬁcation performance of the ensemble is not a trivial result of its good classiﬁcation performance [Table] [Table] [Figure]
Revised: In summary, we show that a selective classiﬁer with a good classiﬁcation performance is not guaranteed to have good selective classiﬁcation performance, so the good selective classiﬁcation performance of the ensemble is not a trivial result of its good classiﬁcation performance.

Original: [Figure] [Figure] coverage 0. s e
Revised: F. T HE E FFECTS OF L ABEL N OISE OF SVHN ON S ELECTIVE C LASSIFIER E NSEMBLES

Original: l e c t i v e r i s k 4 VGG-16 Ensemble
Revised: In this section, we compare the effect of label noise of SVHN on the SR ensemble with that on

Original: Big VGG- In this section, we compare the effect of label noise of SVHN on SR ensemble with that on SAT ensemble, whose result might explain the abnormal experimental results (compared to results on other datasets) on SVHN in Section 6.2.
Revised: SAT ensemble, whose result might explain the abnormal experimental results (compared to results on other datasets) on SVHN in Section 6.2.

Original: SVHN is not a clean dataset, andthere is much more label  noisethat can be detected in SVHN than in CIFAR-10 and CIFAR-100.
Revised: SVHN is not a clean dataset, and much more label [Table] [Table] noise can be detected in SVHN than in CIFAR-10 and CIFAR-100.

Original: Using the soft label of

Original:  SAT (Huang et al., 2020), we detect label noise in SVHN, CIFAR-10, and CIFAR-100, and ﬁnd that
Revised: Using the soft label of SAT (Huang et al., 2020), we detect label noise in SVHN, CIFAR- [Table] and CIFAR- [Table]

Original:  SVHN has signiﬁcantly more label noise than CIFAR-10 and CIFAR-100.
Revised: and ﬁnd that SVHN has signiﬁcantly more label noise than CIFAR-10 and CIFAR-100.

Original: The result is presented in the following.
Revised: The result is presented in the following.

Original: In addition, it is known that SAT is robust to label noise (Huang et al., 2020), while
Revised: In addition, it is known that SAT is robust to label noise (Huang et al., 2020) [Table] while

Original: SR is not so, so we conjecture that the label noise of SVHN isthe reason why the SR ensemble is inferior to SAT on SVHN.
Revised: SR is not so, so we conjecture that the label noise of SVHN is why the SR ensemble is inferior to SAT on SVHN.

Original: We detect label noise with the help of the soft label of SAT.
Revised: We detect label noise with the help of the soft label of SAT.

Original: For a sample x i , the soft label of SAT (Huang et al., 2020), t i,y i , is used to measure x i ’s learning difﬁculty.
Revised: For a sample x i , the soft label of SAT (Huang et al., 2020), t i,y i , is used to measure x i ’s learning difﬁculty.

Original: The soft label of SAT is initialized as 1 and updated at every training epoch as below
Revised: The soft label of SAT is initialized as 1 and updated at every training epoch as below

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Revised: e rr ( % ) g

Revised: i v e n

Revised: c

Revised: o

Revised: v e r a g e o f %

Revised: CIFARSVHN

Revised: CIFARMRPC

Original:  where p θ ( Y | x ) is the predictive probability distribution of the classiﬁer, y i is the label of x i , α is a hyperparameter.
Revised: QNLI MNLI [Figure] where p θ ( Y | x ) is the predictive probability distribution of the classiﬁer, y i is the label of x i , α is a hyperparameter.

Original: The smaller the t i,y i is, the lower the true class predictive probability of the classiﬁer on x i during training time, indicating that x i is more difﬁcult to learn.
Revised: The smaller the t i,y i is, the lower the true class predictive probability of the classiﬁer on x i during training time, indicating that x i is more difﬁcult to learn.

Original: By selecting a percentage of samples with the lowest t i,y i , we get the most difﬁcult samples to learn for the classiﬁer, from which we can easily detect label noise manually.
Revised: By selecting a percentage of samples with the lowest t i,y i , we get the most difﬁcult samples to learn for the classiﬁer, from which we can easily detect label noise manually.

Original: [Table]

Original: In training sets of SVHN, CIFAR-10, and CIFAR-100, we detect label noise manually among the top-0.1% difﬁcult (measured by the soft label of SAT) samples.
Revised: In training sets of SVHN, CIFAR-10, and CIFAR-100, we detect label noise manually among the top-0.1% difﬁcult (measured by the soft label of SAT) samples.

Original: The numbers of mislabeled samples detected in SVHN, CIFAR-10, and CIFAR100 are shown in Table 4.
Revised: The numbers of mislabeled samples detected in SVHN, CIFAR-10, and CIFAR100 are shown in Table 6.

Original: The result shows that
Revised: The result shows that

Original: SVHN has signiﬁcantly more mislabeled samples detected than CIFAR-10 and CIFAR-100, indicating much more label noise in SVHN than  CIFAR-10 and CIFAR-100.
Revised: SVHN has signiﬁcantly more mislabeled samples detected than CIFAR-10 and CIFAR-100, indicating much more label noise in SVHN than in CIFAR-10 and CIFAR-100.

Original: To verify the effect of label noise, the following experiments are designed.
Revised: To verify the effect of label noise, the following experiments are designed.

Original: Firstly, we detect label noise manually among the 1% of the hardest-to-learn samples of SVHN training set and test set, using the soft label of SAT.
Revised: Firstly, we detect label noise manually among the [Table] [Table] [Table]

Revised: coverage 0.

Revised: s e

Revised: l e c

Revised: t

Revised: i v e r

Revised: i s k 4 VGG-16 Ensemble

Revised: Big VGG-

Original:  Secondly, we remove the detected mislabeled samples from the original dataset.
Revised: Dataset [Table] using the soft label of SAT. Secondly, we remove the detected mislabeled samples from the original dataset.

Original: The remaining SVHN dataset is called the clean SVHN .
Revised: The remaining SVHN dataset is called the clean SVHN .

Original: Accordingly, the original dataset is called the original SVHN .
Revised: Accordingly, the original dataset is called the original SVHN .

Original: Finally, we retrain and test the SR ensemble and SAT ensemble and compare their test results.
Revised: Finally, we retrain and test the SR ensemble and SAT ensemble and compare their test results.

Original: In the second step, the reason for removing mislabeled samples rather than modifying them is that some samples cannot be classiﬁed even by humans, and some samples are not in the range of categories of SVHN.
Revised: In the second step, the reason for removing mislabeled samples rather than modifying them is that some samples cannot be classiﬁed even by humans, and some samples are not in the range of categories of SVHN.

Original: Thus, the label noise cannot be eliminated by modifying the labels but by removing mislabeled samples.
Revised: Thus, the label noise cannot be eliminated by modifying the labels but by removing mislabeled samples.

Original: The test results of the SR ensemble and SAT ensemble on clean SVHN are shown in Table 5.
Revised: The test results of the SR ensemble and SAT ensemble on clean SVHN are shown in Table 7.

Original: It is not surprising that the AURCs of the SR ensemble and SAT ensemble are signiﬁcantly lower on the clean SVHN than the original SVHN.
Revised: It is not surprising that the AURCs of the SR ensemble and SAT ensemble are signiﬁcantly lower on the clean SVHN than the original SVHN.

Original: Furthermore, on the clean SVHN, when the number of members is 5, the AURC of the SR ensemble is lower than that of SAT ensemble.
Revised: Furthermore, on the clean SVHN, when the number of members is 5, the AURC of the SR ensemble is lower than that of SAT ensemble.

Original: Combined with results on the original SVHN, where the AURC of  SR ensemble is higher than that of SAT ensemble, we conclude that label noise in SVHN isthe reason why the SR ensemble has a higher AURC than
Revised: Combined with results on the original SVHN, where the AURC of the SR ensemble is higher than that of SAT ensemble, we conclude that label noise in SVHN is why the SR ensemble has a higher AURC than

Original: SAT ensemble.
Revised: SAT ensemble.

Original: In other words, label noise isthe reason why the SR ensemble performs worse in selective classiﬁcation than SAT ensemble on SVHN. 
Revised: In other words, label noise is why the SR ensemble performs worse in selective classiﬁcation than SAT ensemble on SVHN. [Table]

Original: [Table]
Revised: [Table]

Original: In summary, by experiments, we show that the SR ensemble is not as robust to label noise as SAT ensemble, and label noise in SVHN isthe reason why the SR ensemble is not as good as SAT ensemble onSVHN.
Revised: In summary, by experiments, we show that the SR ensemble is not as robust to label noise as SAT ensemble, and label noise in SVHN is why the SR ensemble is not as good as SAT ensemble on

Original:  We construct the clean SVHN , which is SVHN without some mislabeled samples.
Revised: SVHN. We construct the clean SVHN , which is SVHN without some mislabeled samples.

Original: On the clean SVHN, we compare the SR ensemble with SAT ensemble and ﬁnd that the SR ensemble is superior to SAT ensemble in selective classiﬁcation performance.
Revised: On the clean SVHN, we compare the SR ensemble with SAT ensemble and ﬁnd that the SR ensemble is superior to SAT ensemble in selective classiﬁcation performance.

Original: Combined with former experimental results, we conclude that label noise in SVHN isthe reason why  SR ensemble is inferior to 
Revised: Combined with former experimental results, we conclude that label noise in SVHN is why the SR ensemble is inferior to SAT on

Original: SAT on SVHN.
Revised: SVHN.

Original: Considering the experimental results on the clean SVHN and previous experimental results on
Revised: Considering the experimental results on the clean SVHN and previous experimental results on

Original: CIFAR-10 and CIFAR-100 (see Table 5 and Table 1),it can be seen that the SR ensemble is superior to SAT ensemble in selective classiﬁcation on clean image classiﬁcation datasets, Thus, SR ensemble is the state-of-the-art selective classiﬁcation method on clean image classiﬁcation datasets, but is not as robust to label noise as SAT ensemble. 
Revised: CIFAR-10 and CIFAR-100 (see Table 7 and Table 1), the SR ensemble is superior to SAT ensemble in selective classiﬁcation on clean image classiﬁcation datasets, Thus, SR ensemble is the state-ofthe-art selective classiﬁcation method on clean image classiﬁcation datasets, but is not as robust to label noise as SAT ensemble. G A N E XTENSION OF T HEOREM This section discusses the lower bound of φ 0 mentioned in Theorem 2. We aim to calculate φ 0 ’s lower bound without training an ensemble (otherwise, we can measure it directly on the ensemble).

Revised: Preliminary.

Revised: The φ 0 can be obtained by solving the following optimization problem.

Revised: [Equation]

Revised: where R ens ( φ ) and R ( φ ) are the selective risks of the ensemble and the individual model under coverage φ , respectively.

Revised: Suppose that we have all knowledge about the individual model , e.g., the mapping from φ to R is known.

Revised: Since maximizing coverage is equivalent to minimizing the conﬁdence threshold for a ﬁxed model, the original optimization problem can be substituted by

Revised: [Equation]

Revised: where φ ens ( τ ens ) is coverage of the ensemble with conﬁdence threshold τ ens , and φ ( τ ) is coverage of the individual model given the conﬁdence threshold τ , and then φ 0 = φ ( τ ∗ ) , where τ ∗ is the optimal solution to (42).

Revised: Assume that R is a monotone increasing function and that φ is a monotone decreasing function 5 , then it is easy to show, using proof by contradiction, that (42) can be further transformed into

Revised: [Equation]

Revised: To solve (43), we need more information about the ensemble.

Revised: Besides the number of classes K , assume that we know: an oracle that tells whether a sample is deﬁnite; M , the number of member models; and B , the upper bound of p Π k 1 ,..., Π kM ( ·| A ) for all k ∈ { 1 , 2 , ..., K } .

Revised: It is natural to know K and M , and we need the oracle and B because they provide critical information about the ensemble’s behavior.

Revised: The oracle can be implemented by an ensemble with M (cid:48)

Revised: ( M (cid:48) (cid:28) M ) members.

Revised: Eliminating the Unknowns.

Revised: We are now committed to translating the unknowns in the (43) into known quantities.

Revised: Firstly, we eliminate the unknowns in the ﬁrst constraint.

Revised: According to (20) and

Revised: Lemma 4, it is easy to prove (by an integral)

Revised: [Equation]

Revised: 44) where C and C ens are the conﬁdence scores of the individual model and the ensemble, respectively,

Revised: A / D represents the event that the input sample is ambiguous/deﬁnite, β = K · M M − 1 · B .

Revised: Combining

Revised: (44) with φ ens =

Revised: Pr( C ≥ τ ens , A

Revised: [Equation]

Revised: , we have

Revised: [Equation]

Revised: Thus, we can intensify the ﬁrst constraint as

Revised: [Equation]

Revised: Secondly, we eliminate the unknowns in the second constraint.

Revised: according to (41), (35) is a sufﬁcient condition of R ens < R .

Revised: We rewrite (35) as

Revised: [Equation]

Revised: where C ens is the conﬁdence score of the ensemble, Err represents the event that the individual model makes a error prediction.

Revised: Note that the ﬁrst term in the second constraint of (47) contains

Revised: C ens , which is unknown, so we cannot directly replace (43)’s second constraint with (47).

Revised: We eliminate C ens as follows:

Revised: [Equation]

Revised: where the ﬁrst inequality is due to (44), and the second inequality is due to (45) and (46).

Revised: Thus, a sufﬁcient condition of (47) is

Revised: [Equation]

Revised: with which we replace the second constraint of (43).

Revised: In summary, we can intensify the constraints of (43) and obtain the following optimization problem that does not contain the unknowns.

Revised: It is easy to see that optimal solution to (49) is a upper bound of that to (43).

Revised: [Equation]

Revised: Further Simpliﬁcation and Final Result.

Revised: It is easy to show, by proof of contraction, that the ﬁrst constraint of (49) can be substituted by

Revised: [Equation]

Revised: Thus, the second constraint of (49) can be simpliﬁed as

Revised: [Equation]

Revised: ⇔ Pr

Revised: [Equation]

Revised: A )

Revised: [Equation]

Revised: β

Revised: [Equation]

Revised: 1 − τ

Revised: [Equation]

Revised: | D

Revised: [Equation]

Revised: C ≥ τ

Revised: [Equation]

Revised: Pr

Revised: [Equation]

Revised: ≥

Revised: [Equation]

Revised: < R

Revised: [Equation]

Revised: φ

Revised: [Equation]

Revised: ⇔ Pr

Revised: [Equation]

Revised: A )

Revised: [Equation]

Revised: β

Revised: [Equation]

Revised: [Equation]

Revised: τ

Revised: [Equation]

Revised: [Equation]

Revised: Pr( Err

Revised: [Equation]

Revised: C ≥ τ

Revised: [Equation]

Revised: ) <

Revised: R ( φ ( τ )) φ ( τ ) ⇔ Pr( A ) · β (1 − τ ens )

Revised: M + Pr( Err, D, C ≥ τ ens ) <

Revised: Pr( Err, C ≥ τ ) .

Revised: Thus, the ﬁnal version of the optimization problem with respect to τ is

Revised: [Equation]

Revised: ( Err, C ≥ τ ) .

Revised: if τ ens is not None and V ERIFY S ECOND C ONSTRAINT ( τ , τ ens , θ , D , Ω , K , M , B ) is True then right else left =

Revised: τ τ ∗ =

Revised: + right return 1 N

Revised: Ni =1 I {

Revised: ) ≥ τ ∗ } // is the conﬁdence of on sample x i .

Revised: Suppose φ 0 = φ ( τ 0 ) , since τ 0 is the optimal solution to (43), the optimal solution to (50) (denoted as τ ∗ ) provides an upper bound of τ 0 .

Original: {+Thus, considering φ is a monotone decreasing function with respect to τ , φ ( τ ∗ ) is a lower bound of φ 
+}
Revised: {+Thus, considering φ is a monotone decreasing function with respect to τ , φ ( τ ∗ ) is a lower bound of φ 
+}

Revised: = φ ( τ 0 ) .

Revised: Algorithm.

Revised: We design Algorithm 1 to search for the solution to (50) and then obtain the lower bound of φ 0 .

Revised: Since τ ens is determined by τ (the ﬁrst constraint of (50)), (50) can be reduced to a one-dimensional search problem.

Revised: We adopt a binary search in our algorithm for efﬁciency, although this method might provide a suboptimal solution.

Revised: The procedure of Algorithm 1 in each iteration of the binary search is as follows.

Revised: Given current τ , Algorithm 1 determines τ ens using S EARCH F OR T AU E NS (see Algorithm 2), a procedure that searches for τ ens ∈ [0 , 1] using binary search s.t.

Revised: φ ( τ ) =

Revised: Pr( C ≥ τ ens , D ) .

Revised: Note that τ

Revised: ens might not exist, as long as τ is so low

Revised: that φ ( τ ) >

Revised: Pr( D ) = sup

Revised: τ ens Pr( C ≥ τ ens , D ) .

Revised: This problem will be addressed shortly.

Revised: Algorithm 1 exams whether τ ens exists.

Revised: If τ ens exists, Algorithm 1 then exams whether the second constraints of (50) holds for current τ and τ ens , which is implemented by V ERIFYS ECOND C ONSTRAINT (see Algorithm 3).

Revised: If τ ens exists and the second constraint holds, Algorithm 1 searches for a smaller τ in the left half feasible area; otherwise, Algorithm 1 searches for a greater τ in the right half feasible area.

Revised: Once the binary search completes and outputs τ ∗ , Algorithm 1 returns the coverage of θ with conﬁdence threshold τ ∗ .

Revised: An Example.

Revised: To show that Algorithm 1 works in reality, we run this algorithm on CIFAR-10, using the same individual model as Section 6.

Revised: In this example, K = 10 , M = 5 , the oracle is implemented by another ensemble with two individual models (the oracle outputs True if and only if the STD over member models’ predictive distributions < 10 − 3 ).

Revised: Note that it is difﬁcult to estimate of B .

Revised: On the one hand, we need to train an ensemble with M models to estimate B , which is costly.

Revised: On the other hand, the domain of p Π k 1 ,..., Π kM ( ·| A ) has high dimension, so the observed data points are sparse in this domain, which makes the estimation of B more difﬁcult.

Revised: Thus, we do not estimate B but try several hypothetical values of B to see at what B the lower bound of φ 0 is big.

Original: {+With different B s, we obtain different lower bounds of φ 
+}
Revised: {+With different B s, we obtain different lower bounds of φ 
+}

Revised: as Table 8 shows.

Revised: We can see that when B ≤ 10 8 , the lower bound of φ 0 is greater than 50%, which indicates that Algorithm 1 may be robust to the choice of B . is the on x if φ > 1 N (cid:80)

Revised: Ni =1 Ω( x i ) then return None left =

Revised: right

Revised: (cid:15) = 10 −while right − left do τ ens =

Revised: ( left + right

Revised: /if 1 N (cid:80) Ni

Original: {+=
+}
Revised: {+=
+}

Revised: I { C ( x i ; θ ) ≥ τ ens } · Ω( x i ) <

Revised: φ then right

Revised: =

Revised: τ

Revised: ens else left = τ ens return ( left + right ) /

Revised: Input: the conﬁdence threshold τ ; τ ens ; the individual model θ ; the test set D i , y i ) } Ni =1 ; the oracle Ω : X → { 0 , 1 } that tells whether a sample is deﬁnite; the number of classes

Revised: K ; the number of member models M ; and B , the upper bound of p Π k 1 ,..., Π kM ( ·| A ) for all k ∈ { 1 , 2 , ..., K } .

Revised: Output

Revised: P A = 1 N (cid:80) Ni =1 [1 − Ω( x i )]

Revised: β = K · M M − 1 · B leftHandSide = P A · β (1 − τ ens )

Revised: M + 1 N (cid:80)

Revised: Ni

Original: {+=
+}
Revised: {+=
+}

Revised: I { f ( x i ; θ ) (cid:54) =

Revised: y i } · Ω( x i

Revised: [Equation]

Revised: This example also indicates the relationship between ensemble’s diversity and its selective classiﬁcation performance.

Revised: Since an ensemble with a smaller B seems to have more diversity over ambiguous samples, the result in Table 8 suggests that as long as the ensemble has enough diversity over ambiguous samples, the ensemble is guaranteed to have a lower selective risk than the individual model under a considerable range of coverage

Revised: [Table]

Revised: [Table]