Original: S ELECTIVE C LASSIFIER E NSEMBLE
Revised: S ELECTIVE C LASSIFIER E NSEMBLE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Selective classiﬁcation allows a machine learning model to abstain from predicting some hard inputs and thus improve the safety of its predictions.
Revised: Selective classiﬁcation allows a machine learning model to abstain from predicting some hard inputs and thus improve the safety of its predictions.

Original: In this paper, we study the ensemble of selective classiﬁers, i.e. selective classiﬁer ensemble , which combines several weak selective classiﬁers to obtain a more powerful model.
Revised: In this paper, we study the ensemble of selective classiﬁers, i.e. selective classiﬁer ensemble , which combines several weak selective classiﬁers to obtain a more powerful model.

Original: We prove that under some assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.
Revised: We prove that under some assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.

Original: The proof is nontrivial since the selective risk is a non-convex function of the model prediction.
Revised: The proof is nontrivial since the selective risk is a non-convex function of the model prediction.

Original: The assumptions and the theoretical result are supported by systematic experiments on both computer vision and natural language processing tasks.
Revised: The assumptions and the theoretical result are supported by systematic experiments on both computer vision and natural language processing tasks.

Original: A surprising empirical result is that a simple selective classiﬁer ensemble, namely, the ensemble model with maximum probability as conﬁdence , is the state-of-the-art selective classiﬁer.
Revised: A surprising empirical result is that a simple selective classiﬁer ensemble, namely, the ensemble model with maximum probability as conﬁdence , is the state-of-the-art selective classiﬁer.

Original: For instance, on CIFAR-10, using the same VGG-16 backbone model, this ensemble reduces the AURC (Area Under Risk-Coverage Curve) by about 24%, relative to the previous state-of-the-art method.
Revised: For instance, on CIFAR-10, using the same VGG-16 backbone model, this ensemble reduces the AURC (Area Under Risk-Coverage Curve) by about 24%, relative to the previous state-of-the-art method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Although recent years have witnessed the broad applications of deep learning models, their securities have not been fully guaranteed, which gives rise to the study of selective classiﬁcation.
Revised: Although recent years have witnessed the broad applications of deep learning models, their securities have not been fully guaranteed, which gives rise to the study of selective classiﬁcation.

Original: For any given deep learning classiﬁer, there might be inputs that the model is not able to classify in practical applications, for which the model might make unpredictable errors.
Revised: For any given deep learning classiﬁer, there might be inputs that the model is not able to classify in practical applications, for which the model might make unpredictable errors.

Original: To prevent this kind of error, we must accurately delimit the deep learning classiﬁer’s application scope.
Revised: To prevent this kind of error, we must accurately delimit the deep learning classiﬁer’s application scope.

Original: This need gives rise to the study of selective classiﬁcation that learns a selective classiﬁer ( f, g ) , where f is a conventional classiﬁer, and g is a selective function that decides whether the selective classiﬁer should abstain from prediction.
Revised: This need gives rise to the study of selective classiﬁcation that learns a selective classiﬁer ( f, g ) , where f is a conventional classiﬁer, and g is a selective function that decides whether the selective classiﬁer should abstain from prediction.

Original: Since the classiﬁer is well studied, the study of selective classiﬁcation focuses on the design of the selective function.
Revised: Since the classiﬁer is well studied, the study of selective classiﬁcation focuses on the design of the selective function.

Original: A standard approach to designing the selective function is to design a conﬁdence score function with a threshold, and several conﬁdence score functions have been developed.
Revised: A standard approach to designing the selective function is to design a conﬁdence score function with a threshold, and several conﬁdence score functions have been developed.

Original: A simple conﬁdence score function is the maximum predictive probability of the classiﬁer (Hendrycks & Gimpel, 2017).
Revised: A simple conﬁdence score function is the maximum predictive probability of the classiﬁer (Hendrycks & Gimpel, 2017).

Original: More advanced methods modify the model architecture (Geifman & El-Yaniv, 2019) or the loss function (Liu et al., 2019; Huang et al., 2020) of the classiﬁer to train the conﬁdence score function and the classiﬁer simultaneously.
Revised: More advanced methods modify the model architecture (Geifman & El-Yaniv, 2019) or the loss function (Liu et al., 2019; Huang et al., 2020) of the classiﬁer to train the conﬁdence score function and the classiﬁer simultaneously.

Original: For example, Deep Gambler (Liu et al., 2019) regards the selective classiﬁcation problem as gambling and proposes a novel loss function to train the classiﬁer and the conﬁdence score function.
Revised: For example, Deep Gambler (Liu et al., 2019) regards the selective classiﬁcation problem as gambling and proposes a novel loss function to train the classiﬁer and the conﬁdence score function.

Original: Although there are various individual models for the selective classiﬁer, there has been no systematic study of the ensemble method in selective classiﬁcation.
Revised: Although there are various individual models for the selective classiﬁer, there has been no systematic study of the ensemble method in selective classiﬁcation.

Original: It is well known that the ensemble method, which combines the individual models to obtain a more powerful model, can improve the predictive performances of machine learning models (see Zhou (2012) for a review), but only a particular selective classiﬁer ensemble, the ensemble of Softmax
Revised: It is well known that the ensemble method, which combines the individual models to obtain a more powerful model, can improve the predictive performances of machine learning models (see Zhou (2012) for a review), but only a particular selective classiﬁer ensemble, the ensemble of Softmax

Original: Response (Hendrycks & Gimpel, 2017), has been empirically studied by Lakshminarayanan et al.
Revised: Response (Hendrycks & Gimpel, 2017), has been empirically studied by Lakshminarayanan et al.

Original: Ensembles of other kinds of selective classiﬁers, and the theoretical foundation of the ensemble in selective classiﬁcation have not been studied yet.
Revised: Ensembles of other kinds of selective classiﬁers, and the theoretical foundation of the ensemble in selective classiﬁcation have not been studied yet.

Original: In this paper, we ﬁrst demonstrate the theoretical foundation of the ensemble on selective classiﬁers, that is, with some assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.
Revised: In this paper, we ﬁrst demonstrate the theoretical foundation of the ensemble on selective classiﬁers, that is, with some assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.

Original: The proof is nontrivial since the selective risk (with the 0/1 loss) are non-convex.
Revised: The proof is nontrivial since the selective risk (with the 0/1 loss) are non-convex.

Original: Second, we show the experimental results of the ensemble’s performance in selective classiﬁcation.
Revised: Second, we show the experimental results of the ensemble’s performance in selective classiﬁcation.

Original: The contributions of this paper are summarized as follows.
Revised: The contributions of this paper are summarized as follows.

Original: • We are the ﬁrst to theoretically demonstrate that based on several reasonable assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.
Revised: • We are the ﬁrst to theoretically demonstrate that based on several reasonable assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.

Original: We verify this by systematic experiments on the tasks of image classiﬁcation and text classiﬁcation.
Revised: We verify this by systematic experiments on the tasks of image classiﬁcation and text classiﬁcation.

Original: • We show a surprising experimental result that two simple methods, the SR ensemble and the Reg-curr ensemble, which can be summarized as the ensemble model with maximum probability as conﬁdence , are the state-of-the-art selective classiﬁers.
Revised: • We show a surprising experimental result that two simple methods, the SR ensemble and the Reg-curr ensemble, which can be summarized as the ensemble model with maximum probability as conﬁdence , are the state-of-the-art selective classiﬁers.

Original:  P ROBLEM F ORMULATION OF S ELECTIVE C LASSIFICATION
Revised:  P ROBLEM F ORMULATION OF S ELECTIVE C LASSIFICATION

Original: A selective classiﬁer is composed of a standard classiﬁer and a selective function.
Revised: A selective classiﬁer is composed of a standard classiﬁer and a selective function.

Original: Considering a standard classiﬁcation problem, X is a feature space, Y = { 1 , 2 , ..., K } is a ﬁnite label set, and a classiﬁer f is a function f : X → Y .
Revised: Considering a standard classiﬁcation problem, X is a feature space, Y = { 1 , 2 , ..., K } is a ﬁnite label set, and a classiﬁer f is a function f : X → Y .

Original: A labeled dataset D = { ( x i , y i ) } Ni =0 ⊆
Revised: A labeled dataset D = { ( x i , y i ) } Ni =0 ⊆

Original: X × Y is sampled from a distribution p X , Y .
Revised: X × Y is sampled from a distribution p X , Y .

Original: Our goal is to learn a selective classiﬁer where f is a standard classiﬁer and g : X → { 0 , 1 } is a selective function that estimates the correctness of f ’s prediction.
Revised: Our goal is to learn a selective classiﬁer where f is a standard classiﬁer and g : X → { 0 , 1 } is a selective function that estimates the correctness of f ’s prediction.

Original: Given input x , the output of selective classiﬁer ( f, g ) is ( f, g )( x ) = (cid:26) f ( x ) , if g ( x ) =
Revised: Given input x , the output of selective classiﬁer ( f, g ) is ( f, g )( x ) = (cid:26) f ( x ) , if g ( x ) =

Original: Abstain , if g ( x ) = 0 .
Revised: Abstain , if g ( x ) = 0 .

Original: Usually, g is realized by a conﬁdence score ˆ κ : X → R + with a threshold τ (Geifman & El-Yaniv, 2017), namely g ( x ) =
Revised: Usually, g is realized by a conﬁdence score ˆ κ : X → R + with a threshold τ (Geifman & El-Yaniv, 2017), namely g ( x ) =

Original: I { ˆ κ ( x ) > τ } , (2) where I is the indicator function.
Revised: I { ˆ κ ( x ) > τ } , (2) where I is the indicator function.

Original: Coverage and selective risk are two basic evaluation metrics of selective classiﬁers, and the goal of selective classiﬁers is to minimize the selective risk for target coverage.
Revised: Coverage and selective risk are two basic evaluation metrics of selective classiﬁers, and the goal of selective classiﬁers is to minimize the selective risk for target coverage.

Original: The coverage of ( f, g ) is deﬁned to be the probability of ( f, g ) not abstaining from prediction (Geifman & El-Yaniv, 2017), i.e.
Revised: The coverage of ( f, g ) is deﬁned to be the probability of ( f, g ) not abstaining from prediction (Geifman & El-Yaniv, 2017), i.e.

Original: [Equation]
Revised: [Equation]

Original: where p ( x ) is the probability density function of input x .
Revised: where p ( x ) is the probability density function of input x .

Original: The selective risk (Geifman & El-Yaniv, 2017) of ( f, g ) is
Revised: The selective risk (Geifman & El-Yaniv, 2017) of ( f, g ) is

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) :
Revised: where (cid:96) :

Original: Y × Y → R + is a given loss function.
Revised: Y × Y → R + is a given loss function.

Original: Usually, (cid:96) is the 0/1 loss (Geifman & El-Yaniv, 2017; 2019;
Revised: Usually, (cid:96) is the 0/1 loss (Geifman & El-Yaniv, 2017; 2019;

Original: Liu et al., 2019; Huang et al., 2020).
Revised: Liu et al., 2019; Huang et al., 2020).

Original: Based on these deﬁnitions, the objective of selective classiﬁers is formalized as
Revised: Based on these deﬁnitions, the objective of selective classiﬁers is formalized as

Original: [Equation]
Revised: [Equation]

Original: where c target is a given target coverage.
Revised: where c target is a given target coverage.

Original: When the selective function g is developed as (2), the conﬁdence threshold τ controls the tradeoff between coverage and selective risk.
Revised: When the selective function g is developed as (2), the conﬁdence threshold τ controls the tradeoff between coverage and selective risk.

Original: With different values of τ , ( f, g ) has different pairs of coverage and selective risk ( φ ( f, g ; τ ) , R ( f, g ; τ )) , which forms the risk-coverage curve (Geifman & El-Yaniv, 2017) of ( f, g ) .
Revised: With different values of τ , ( f, g ) has different pairs of coverage and selective risk ( φ ( f, g ; τ ) , R ( f, g ; τ )) , which forms the risk-coverage curve (Geifman & El-Yaniv, 2017) of ( f, g ) .

Original: The risk-coverage curve speciﬁes the entire performance proﬁle of a selective classiﬁer, and it is easy to see that the selective classiﬁer with a lower risk-coverage curve is better.
Revised: The risk-coverage curve speciﬁes the entire performance proﬁle of a selective classiﬁer, and it is easy to see that the selective classiﬁer with a lower risk-coverage curve is better.

Original: To evaluate selective classiﬁers more concisely, the area under the risk-coverage curve ( AURC ) is introduced as a metric of selective classiﬁers (Xin et al., 2021), and the selective classiﬁer with a lower AURC is better.
Revised: To evaluate selective classiﬁers more concisely, the area under the risk-coverage curve ( AURC ) is introduced as a metric of selective classiﬁers (Xin et al., 2021), and the selective classiﬁer with a lower AURC is better.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Here, we summarize the previous studies on selective classiﬁcation and ensemble methods.
Revised: Here, we summarize the previous studies on selective classiﬁcation and ensemble methods.

Original: We also discuss the difference between selective classiﬁcation and out-of-distribution detection.
Revised: We also discuss the difference between selective classiﬁcation and out-of-distribution detection.

Original: 3. S ELECTIVE C LASSIFICATION
Revised: 3. S ELECTIVE C LASSIFICATION

Original: The critical problem of designing a selective classiﬁer is to design its selective function, and there are two types of selective function g .
Revised: The critical problem of designing a selective classiﬁer is to design its selective function, and there are two types of selective function g .

Original: One is the implicit selective function , which is derived from the classiﬁer.
Revised: One is the implicit selective function , which is derived from the classiﬁer.

Original: The other is the explicit selective function , a neural network trained with the classiﬁer f simultaneously.
Revised: The other is the explicit selective function , a neural network trained with the classiﬁer f simultaneously.

Original: Previous works for selective classiﬁers are listed as follows.
Revised: Previous works for selective classiﬁers are listed as follows.

Original: Selective classiﬁers with implicit selective functions include SR (Softmax Response) (Hendrycks & Gimpel, 2017), MC-Dropout (Monte Carlo-Dropout) (Gal & Ghahramani, 2016), and Reg-curr (Xin et al., 2021).
Revised: Selective classiﬁers with implicit selective functions include SR (Softmax Response) (Hendrycks & Gimpel, 2017), MC-Dropout (Monte Carlo-Dropout) (Gal & Ghahramani, 2016), and Reg-curr (Xin et al., 2021).

Original: In SR, the selective classiﬁer is a vanilla classiﬁer with its maximum predictive probability as conﬁdence (i.e., the maximum output of the softmax layer).
Revised: In SR, the selective classiﬁer is a vanilla classiﬁer with its maximum predictive probability as conﬁdence (i.e., the maximum output of the softmax layer).

Original: In MC-Dropout, it enables the dropout layer of the classiﬁer and runs multiple feed-forward iterations at inference time to obtain the variance of the maximum probability output of f ’s softmax layer, whose negative value is used as the conﬁdence score.
Revised: In MC-Dropout, it enables the dropout layer of the classiﬁer and runs multiple feed-forward iterations at inference time to obtain the variance of the maximum probability output of f ’s softmax layer, whose negative value is used as the conﬁdence score.

Original: Reg-curr behaves the same as the SR at inference time but uses an RPPbased regularizer at training time, where RPP (Reversed Pair Proportion) (Xin et al., 2021) is the proportion of reversed pairs of conﬁdence scores.
Revised: Reg-curr behaves the same as the SR at inference time but uses an RPPbased regularizer at training time, where RPP (Reversed Pair Proportion) (Xin et al., 2021) is the proportion of reversed pairs of conﬁdence scores.

Original: Selective classiﬁers with explicit selective functions include SN (SelectiveNet) (Geifman & ElYaniv, 2019), Gambler (Deep Gambler)
Revised: Selective classiﬁers with explicit selective functions include SN (SelectiveNet) (Geifman & ElYaniv, 2019), Gambler (Deep Gambler)

Original: (Liu et al., 2019), and SAT (Self-Adaptive Training) (Huang et al., 2020).
Revised: (Liu et al., 2019), and SAT (Self-Adaptive Training) (Huang et al., 2020).

Original: SN is a neural network that combines f and g , where f and g share convolutional layers and have their separate fully-connected layers.
Revised: SN is a neural network that combines f and g , where f and g share convolutional layers and have their separate fully-connected layers.

Original: The loss function of the model is the selective risk with some regularizers.
Revised: The loss function of the model is the selective risk with some regularizers.

Original: A hyperparameter c is needed to specify the target coverage.
Revised: A hyperparameter c is needed to specify the target coverage.

Original: Gambler adds the abstention option to the classiﬁer as an extra class, that is, for a given input x , the predictive probability of the extra class is the conﬁdence of abstention, i.e., 1 − ˆ κ ( x ) .
Revised: Gambler adds the abstention option to the classiﬁer as an extra class, that is, for a given input x , the predictive probability of the extra class is the conﬁdence of abstention, i.e., 1 − ˆ κ ( x ) .

Original: At training time, it regards the selective classiﬁcation problem as gambling and is trained to maximize the gambling reward.
Revised: At training time, it regards the selective classiﬁcation problem as gambling and is trained to maximize the gambling reward.

Original: Similar to Gambler, SAT adds the abstention option to the classiﬁer as an extra class.
Revised: Similar to Gambler, SAT adds the abstention option to the classiﬁer as an extra class.

Original: However, SAT has a different training procedure, which is trained with a soft label that tells the model which sample to reject.
Revised: However, SAT has a different training procedure, which is trained with a soft label that tells the model which sample to reject.

Original: SAT is the previous state-of-the-art selective classiﬁer for image classiﬁcation tasks.
Revised: SAT is the previous state-of-the-art selective classiﬁer for image classiﬁcation tasks.

Original: 3. R ANDOMIZATION - BASED E NSEMBLE
Revised: 3. R ANDOMIZATION - BASED E NSEMBLE

Original: In the randomization-based ensemble method, each member model is trained independently with the same architecture and training procedure but with different randomization seeds for the random initialization of parameters and the random shufﬂing of training data for each training epoch.
Revised: In the randomization-based ensemble method, each member model is trained independently with the same architecture and training procedure but with different randomization seeds for the random initialization of parameters and the random shufﬂing of training data for each training epoch.

Original: At inference time, the predictive probability of the ensemble for each class is the average of those of member models.
Revised: At inference time, the predictive probability of the ensemble for each class is the average of those of member models.

Original: Lakshminarayanan et al.
Revised: Lakshminarayanan et al.

Original: (2017) applies this ensemble method to deep neural networks (Deep Ensemble) and has achieved state-of-the-art performance in uncertainty estimation.
Revised: (2017) applies this ensemble method to deep neural networks (Deep Ensemble) and has achieved state-of-the-art performance in uncertainty estimation.

Original: The difference between Lakshminarayanan et al.
Revised: The difference between Lakshminarayanan et al.

Original: and our work is that they only empirically study the ensemble of vanilla classiﬁers (the SR ensemble), but we not only empirically study ensembles of multiple outstanding selective classiﬁers but also provide the theoretical foundation of the ensemble in selective classiﬁcation.
Revised: and our work is that they only empirically study the ensemble of vanilla classiﬁers (the SR ensemble), but we not only empirically study ensembles of multiple outstanding selective classiﬁers but also provide the theoretical foundation of the ensemble in selective classiﬁcation.

Original: As for the theoretical works, Krogh & Vedelsby (1994) proposes the error-ambiguity decomposition to explain the better performance of the randomizationbased ensemble in regression tasks.
Revised: As for the theoretical works, Krogh & Vedelsby (1994) proposes the error-ambiguity decomposition to explain the better performance of the randomizationbased ensemble in regression tasks.

Original: However, for classiﬁcation tasks, there is no such simple and elegant analysis, since the evaluation metrics are non-convex (Zhou, 2021).
Revised: However, for classiﬁcation tasks, there is no such simple and elegant analysis, since the evaluation metrics are non-convex (Zhou, 2021).

Original: Thus, the corresponding analysis for classiﬁcation tasks needs additional assumptions, e.g., unbiased, uncorrelated, and identically distributed estimation errors for the posterior probability distribution (Tumer & Ghosh, 1996;
Revised: Thus, the corresponding analysis for classiﬁcation tasks needs additional assumptions, e.g., unbiased, uncorrelated, and identically distributed estimation errors for the posterior probability distribution (Tumer & Ghosh, 1996;

Original: Fumera & Roli, 2005).
Revised: Fumera & Roli, 2005).

Original: Nevertheless, these assumptions are impractical (Fumera & Roli, 2005).
Revised: Nevertheless, these assumptions are impractical (Fumera & Roli, 2005).

Original: As far as we know, there is no systematic study of the ensemble in the context of selective classiﬁcation.
Revised: As far as we know, there is no systematic study of the ensemble in the context of selective classiﬁcation.

Original: 3. O UT - OF -D ISTRIBUTION D ETECTION
Revised: 3. O UT - OF -D ISTRIBUTION D ETECTION

Original: A related topic of selective classiﬁcation is out-of-distribution (OOD) detection (Lakshminarayanan et al., 2017))
Revised: A related topic of selective classiﬁcation is out-of-distribution (OOD) detection (Lakshminarayanan et al., 2017))

Original: (also called as open set recognition (Scheirer et al., 2012), or novelty detection (Sch ¨ olkopf et al., 2001)), which detects samples that differ signiﬁcantly from a given dataset, i.e.,
Revised: (also called as open set recognition (Scheirer et al., 2012), or novelty detection (Sch ¨ olkopf et al., 2001)), which detects samples that differ signiﬁcantly from a given dataset, i.e.,

Original: OOD samples.
Revised: OOD samples.

Original: The essential difference between selective classiﬁcation and OOD detection lies in their different goals.
Revised: The essential difference between selective classiﬁcation and OOD detection lies in their different goals.

Original: The goal of the former is to detect samples where the classiﬁer predicts incorrectly, which depends on both the classiﬁer and samples, while that of the latter is to detect samples that differ signiﬁcantly from a given dataset, which depends on samples only.
Revised: The goal of the former is to detect samples where the classiﬁer predicts incorrectly, which depends on both the classiﬁer and samples, while that of the latter is to detect samples that differ signiﬁcantly from a given dataset, which depends on samples only.

Original: In addition, at present, selective classiﬁcation assumes that test data and training data are sampled from the same distribution (Geifman & El-Yaniv, 2017), instead of using OOD test data as OOD detection.
Revised: In addition, at present, selective classiﬁcation assumes that test data and training data are sampled from the same distribution (Geifman & El-Yaniv, 2017), instead of using OOD test data as OOD detection.

Original: Thus, selective classiﬁcation and OOD detection are complementary in preventing error predictions of machine learning models, as Figure 3 shows.
Revised: Thus, selective classiﬁcation and OOD detection are complementary in preventing error predictions of machine learning models, as Figure 3 shows.

Original:  M ETHOD
Revised:  M ETHOD

Original: With the randomization-based ensemble method, we propose the selective classiﬁer ensemble .
Revised: With the randomization-based ensemble method, we propose the selective classiﬁer ensemble .

Original: The basic idea is that each predictive probability (as well as the conﬁdence score in the case of explicit selective functions) of the ensemble should be the average of those of the member models.
Revised: The basic idea is that each predictive probability (as well as the conﬁdence score in the case of explicit selective functions) of the ensemble should be the average of those of the member models.

Original: Formally, we assume that for an input sample x , a classiﬁer f at ﬁrst provides the predictive probability distribution ˆ π θ =
Revised: Formally, we assume that for an input sample x , a classiﬁer f at ﬁrst provides the predictive probability distribution ˆ π θ =

Original: (ˆ π 1 θ , · · · , ˆ π Kθ ) and then makes prediction f
Revised: (ˆ π 1 θ , · · · , ˆ π Kθ ) and then makes prediction f

Original: ( x ; θ ) = arg max 1 ≤ k ≤ K ˆ π kθ ( x ) , where
Revised: ( x ; θ ) = arg max 1 ≤ k ≤ K ˆ π kθ ( x ) , where

Original: K is the number of classes, θ denotes the parameters of f , and ˆ π kθ ( x ) is the predictive probability for class k (the superscript is not an exponent).
Revised: K is the number of classes, θ denotes the parameters of f , and ˆ π kθ ( x ) is the predictive probability for class k (the superscript is not an exponent).

Original: Then, the predictive probability distribution of the ensemble classiﬁer of M member models is
Revised: Then, the predictive probability distribution of the ensemble classiﬁer of M member models is

Original: [Equation]
Revised: [Equation]

Original: The ensemble of the selective function is deﬁned as follows.
Revised: The ensemble of the selective function is deﬁned as follows.

Original: For implicit selective functions (e.g.
Revised: For implicit selective functions (e.g.

Original: SR), to keep the ensemble the same kind of selective classiﬁer as the individual model (for example, the ensemble of SR should still be an SR model), the conﬁdence score of the ensemble is derived from ˆ π ens in the same way as the individual model.
Revised: SR), to keep the ensemble the same kind of selective classiﬁer as the individual model (for example, the ensemble of SR should still be an SR model), the conﬁdence score of the ensemble is derived from ˆ π ens in the same way as the individual model.

Original: For example, the conﬁdence score of the SR ensemble is
Revised: For example, the conﬁdence score of the SR ensemble is

Original: [Equation]
Revised: [Equation]

Original: For explicit selective functions (e.g. SAT), the conﬁdence score of the ensemble is the average of those of member models,
Revised: For explicit selective functions (e.g. SAT), the conﬁdence score of the ensemble is the average of those of member models,

Original: [Equation]
Revised: [Equation]

Original:  T HEORETICAL A NALYSIS OF S ELECTIVE C LASSIFIER E NSEMBLE
Revised:  T HEORETICAL A NALYSIS OF S ELECTIVE C LASSIFIER E NSEMBLE

Original: In this section, we analyze the selective risk (with the 0/1 loss) of the ensemble of a simple selective classiﬁer, the SR ensemble (see Section 4 for its deﬁnition).
Revised: In this section, we analyze the selective risk (with the 0/1 loss) of the ensemble of a simple selective classiﬁer, the SR ensemble (see Section 4 for its deﬁnition).

Original: If the selective risk is a convex function of the predictive probability distribution, then according to the deﬁnition of the convex function, (6) implies that the selective risk of the ensemble is less than or equal to that of the individual model.
Revised: If the selective risk is a convex function of the predictive probability distribution, then according to the deﬁnition of the convex function, (6) implies that the selective risk of the ensemble is less than or equal to that of the individual model.

Original: However, the selective risk is non-convex because the 0/1 loss is a step function.
Revised: However, the selective risk is non-convex because the 0/1 loss is a step function.

Original: Thus, the analysis is not easy.
Revised: Thus, the analysis is not easy.

Original: We need some assumptions to prove a lower selective risk of the ensemble.
Revised: We need some assumptions to prove a lower selective risk of the ensemble.

Original: We introduce the assumptions in Section 5.1 (veriﬁed in Section 6.1) and the theoretical results in Section 5.2.
Revised: We introduce the assumptions in Section 5.1 (veriﬁed in Section 6.1) and the theoretical results in Section 5.2.

Original: The analysis for the other selective classiﬁers is left for future study.
Revised: The analysis for the other selective classiﬁers is left for future study.

Original: 5. A SSUMPTIONS
Revised: 5. A SSUMPTIONS

Original: Given an SR ensemble with M ( M > 1) members, we assume that there are samples on which all member models provide almost the same predictive probability distributions.
Revised: Given an SR ensemble with M ( M > 1) members, we assume that there are samples on which all member models provide almost the same predictive probability distributions.

Original: Furthermore, we idealize them as deﬁnite samples , for which all member models provide precisely the same predictive probability distribution.
Revised: Furthermore, we idealize them as deﬁnite samples , for which all member models provide precisely the same predictive probability distribution.

Original: Then, the rest samples are referred to as ambiguous samples .
Revised: Then, the rest samples are referred to as ambiguous samples .

Original: Let D be the event that the input sample is deﬁnite and A be the event that the input sample is ambiguous.
Revised: Let D be the event that the input sample is deﬁnite and A be the event that the input sample is ambiguous.

Original: Considering that the input sample is randomly drawn from a dataset, the predictive probability for class k (1 ≤ k ≤ K ) and the conﬁdence of the SR model are random variables.
Revised: Considering that the input sample is randomly drawn from a dataset, the predictive probability for class k (1 ≤ k ≤ K ) and the conﬁdence of the SR model are random variables.

Original: We denote these random variables as Π k and C respectively and use π k and κ to denote their values respectively.
Revised: We denote these random variables as Π k and C respectively and use π k and κ to denote their values respectively.

Original: Generally, for any continuous random variable Z, p Z denotes the probability density function ( PDF ) of Z, and for any real variable z , z → 1 − denotes that z approaches 1 from the left.
Revised: Generally, for any continuous random variable Z, p Z denotes the probability density function ( PDF ) of Z, and for any real variable z , z → 1 − denotes that z approaches 1 from the left.

Original: Based on the idealization and notations above, we introduce the following assumptions.
Revised: Based on the idealization and notations above, we introduce the following assumptions.

Original: Assumption 1.
Revised: Assumption 1.

Original: For any individual SR model, with its conﬁdence score denoted as C, we have
Revised: For any individual SR model, with its conﬁdence score denoted as C, we have

Original: [Equation]
Revised: [Equation]

Original: where Err is the event that the model makes an error prediction.
Revised: where Err is the event that the model makes an error prediction.

Original: In Assumption 1, Pr( Err | A, C ≥ τ ) is the selective risk of the individual model with a conﬁdence threshold of τ for ambiguous samples, and Pr( Err | D, C ≥ τ ) is that for deﬁnite samples.
Revised: In Assumption 1, Pr( Err | A, C ≥ τ ) is the selective risk of the individual model with a conﬁdence threshold of τ for ambiguous samples, and Pr( Err | D, C ≥ τ ) is that for deﬁnite samples.

Original: The motivation for Assumption 1 is that ambiguous samples seem more difﬁcult to classify than deﬁnite samples.
Revised: The motivation for Assumption 1 is that ambiguous samples seem more difﬁcult to classify than deﬁnite samples.

Original: Assumption 2.
Revised: Assumption 2.

Original: For any individual SR model, both lim κ → 1 − p C ( κ | D ) and lim κ → 1 − p C
Revised: For any individual SR model, both lim κ → 1 − p C ( κ | D ) and lim κ → 1 − p C

Original: ( κ | A ) exist and are non-zero, i.e., positive, where C is the conﬁdence score of the SR model.
Revised: ( κ | A ) exist and are non-zero, i.e., positive, where C is the conﬁdence score of the SR model.

Original: Assumption 2 claims that for both ambiguous samples and deﬁnite samples, when the conﬁdence score of the individual model approaches 1, its PDF approaches a non-zero value.
Revised: Assumption 2 claims that for both ambiguous samples and deﬁnite samples, when the conﬁdence score of the individual model approaches 1, its PDF approaches a non-zero value.

Original: In other words, the individual model is not modest over both ambiguous samples and deﬁnite samples.
Revised: In other words, the individual model is not modest over both ambiguous samples and deﬁnite samples.

Original: Assumption 3.
Revised: Assumption 3.

Original: For any SR ensemble of M member models, let Π ki be the predictive probability of the i -th member model for class k .
Revised: For any SR ensemble of M member models, let Π ki be the predictive probability of the i -th member model for class k .

Original: Then ∀ k ∈ { 1 , . .
Revised: Then ∀ k ∈ { 1 , . .

Original: , K } , p Π k 1 , Π k 2 ,..., Π kM ( ·| A ) , the joint probability density function of Π k 1 , Π k 2 , . . .
Revised: , K } , p Π k 1 , Π k 2 ,..., Π kM ( ·| A ) , the joint probability density function of Π k 1 , Π k 2 , . . .

Original: , Π kM given the input sample being ambiguous, is bounded.
Revised: , Π kM given the input sample being ambiguous, is bounded.

Original: Assumption 3 is related to the deﬁnition of ambiguous samples and can be understood via the following example.
Revised: Assumption 3 is related to the deﬁnition of ambiguous samples and can be understood via the following example.

Original: Consider an ensemble with two members θ 1 and θ 2 .
Revised: Consider an ensemble with two members θ 1 and θ 2 .

Original: Let their predictive probability for class k be Π k 1 and Π k 2 , k ∈ { 1 , 2 , ..., K } , respectively.
Revised: Let their predictive probability for class k be Π k 1 and Π k 2 , k ∈ { 1 , 2 , ..., K } , respectively.

Original: On deﬁnite samples, both θ 1 and θ 2 provide the same predictive probability distribution ( Π k 1 =
Revised: On deﬁnite samples, both θ 1 and θ 2 provide the same predictive probability distribution ( Π k 1 =

Original: Π k 2 , ∀ k ∈ { 1 , 2 , ..., K } ).
Revised: Π k 2 , ∀ k ∈ { 1 , 2 , ..., K } ).

Original: Thus, for all k , p Π k 1 , Π k 2 ( ·| D ) , the joint distribution of Π k 1 and Π k 2 given the input samples being deﬁnite, collapses to { ( λ, λ ) | λ ∈ [0 , 1] } .
Revised: Thus, for all k , p Π k 1 , Π k 2 ( ·| D ) , the joint distribution of Π k 1 and Π k 2 given the input samples being deﬁnite, collapses to { ( λ, λ ) | λ ∈ [0 , 1] } .

Original: In other words
Revised: In other words,

Revised: [Equation]

Revised: otherwise

Original: [Equation]
Revised: [Equation]

Original: On the contrary, ambiguous samples do not have such  property.
Revised: On the contrary, ambiguous samples do not have such a property.

Original: We intensify this by Assumption 3 to provide a good analytical property of ambiguous samples.
Revised: We intensify this by Assumption 3 to provide a good analytical property of ambiguous samples.

Original: Furthermore, Assumption 3 reﬂects the diversity of the ensemble over ambiguous samples.
Revised: Furthermore, Assumption 3 reﬂects the diversity of the ensemble over ambiguous samples.

Original: Still consider the example above.
Revised: Still consider the example above.

Original: If the predictions of θ 1 and θ 2 are sure to coincide, i.e., the ensemble model has no diversity, then the PDF of Π k 1 and Π k 2 is unbounded.
Revised: If the predictions of θ 1 and θ 2 are sure to coincide, i.e., the ensemble model has no diversity, then the PDF of Π k 1 and Π k 2 is unbounded.

Original: Conversely, if the PDF of Π k 1 and Π k 2 is bounded, then the predictions of the member models are diverse.
Revised: Conversely, if the PDF of Π k 1 and Π k 2 is bounded, then the predictions of the member models are diverse.

Original: Thus, Assumption 3 provides the diversity of the ensemble over ambiguous samples.
Revised: Thus, Assumption 3 provides the diversity of the ensemble over ambiguous samples.

Original: It is well known that the randomization-based ensemble has diversity (Zhou, 2012).
Revised: It is well known that the randomization-based ensemble has diversity (Zhou, 2012).

Original: Since the ensemble does not have diversity over deﬁnite samples, it must have diversity over ambiguous sample.
Revised: Since the ensemble does not have diversity over deﬁnite samples, it must have diversity over ambiguous samples.

Original: Thus, we do not provide experimental veriﬁcation for Assumption 3.
Revised: Thus, we do not provide experimental veriﬁcation for Assumption 3.

Original: 5. A NALYSIS R ESULTS
Revised: 5. A NALYSIS R ESULTS

Original: With the assumptions above, we derive Theorem 2 (see Appendix B for proof details), which shows that the selective risk of the ensemble is lower than that of the individual model under a range of coverage.
Revised: With the assumptions above, we derive Theorem 2 (see Appendix B for proof details), which shows that the selective risk of the ensemble is lower than that of the individual model under a range of coverage.

Original: The intuition of its proof is as follows.
Revised: The intuition of its proof is as follows.

Original: According to Assumption 1, the individual model is not modest over ambiguous samples.
Revised: According to Assumption 1, the individual model is not modest over ambiguous samples.

Original: On the contrary, based on Assumption 3, we prove that the ensemble is modest over ambiguous samples (Proposition 1).
Revised: On the contrary, based on Assumption 3, we prove that the ensemble is modest over ambiguous samples (Proposition 1).

Original: In addition, both the individual model and the ensemble are not modest over deﬁnite samples (due to Assumption 2 and the deﬁnition of deﬁnite samples).
Revised: In addition, both the individual model and the ensemble are not modest over deﬁnite samples (due to Assumption 2 and the deﬁnition of deﬁnite samples).

Original: Thus, considering that the classiﬁer’s error rate over ambiguous samples is higher than that over deﬁnite samples when conﬁdence approaches 1 (Assumption 1), the individual model suffers more wrong but conﬁdent predictions that come with ambiguous samples than the ensemble.
Revised: Thus, considering that the classiﬁer’s error rate over ambiguous samples is higher than that over deﬁnite samples when conﬁdence approaches 1 (Assumption 1), the individual model suffers more wrong but conﬁdent predictions that come with ambiguous samples than the ensemble.

Original: Therefore, the selective risk of the individual model is higher than that of the ensemble (Theorem 2).
Revised: Therefore, the selective risk of the individual model is higher than that of the ensemble (Theorem 2).

Original: In a word, the intuition is that because the ensemble avoids to be overconﬁdent over ambiguous samples, the ensemble has a lower selective risk.
Revised: In a word, the intuition is that because the ensemble avoids to be overconﬁdent over ambiguous samples, the ensemble has a lower selective risk.

Original: Before Theorem 2, we discuss Proposition 1, which provides critical insight into the better performance of the ensemble.
Revised: Before Theorem 2, we discuss Proposition 1, which provides critical insight into the better performance of the ensemble.

Original: Proposition 1.
Revised: Proposition 1.

Original: If Assumption 3 holds, then lim κ ens → 1 − p C ens ( κ ens | A )
Revised: If Assumption 3 holds, then lim κ ens → 1 − p C ens ( κ ens | A )

Original: = 0 , where C ens is the conﬁdence score of the ensemble.
Revised: = 0 , where C ens is the conﬁdence score of the ensemble.

Original: Proposition 1 suggests that given the input sample being ambiguous, when the conﬁdence of the ensemble approaches one, its PDF approaches zero.
Revised: Proposition 1 suggests that given the input sample being ambiguous, when the conﬁdence of the ensemble approaches one, its PDF approaches zero.

Original: In other words, the ensemble is modest over ambiguous samples.
Revised: In other words, the ensemble is modest over ambiguous samples.

Original: Based on this proposition, we prove that the SR ensemble has a lower selective risk than an SR individual model under a range of coverage.
Revised: Based on this proposition, we prove that the SR ensemble has a lower selective risk than an SR individual model under a range of coverage.

Original: Theorem 2.
Revised: Theorem 2.

Original: If Assumption 1-3 holds, then for any individual SR model and any SR ensemble, ∃ φ 0 ∈ (0 , 1) such that ∀ φ ∈ (0 , φ 0 ) ,
Revised: If Assumption 1-3 holds, then for any individual SR model and any SR ensemble, ∃ φ 0 ∈ (0 , 1) such that ∀ φ ∈ (0 , φ 0 ) ,

Original: R ens ( φ ) <
Revised: R ens ( φ ) <

Original: R ind ( φ ) , (9) where R ens ( φ ) and R ind ( φ ) are the selective risks of the SR ensemble and the individual SR model under coverage φ , respectively 1 .
Revised: R ind ( φ ) , (9) where R ens ( φ ) and R ind ( φ ) are the selective risks of the SR ensemble and the individual SR model under coverage φ , respectively 1 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Datasets.
Revised: Datasets.

Original: We conduct experiments on multiple datasets for image classiﬁcation and text classiﬁcation tasks.
Revised: We conduct experiments on multiple datasets for image classiﬁcation and text classiﬁcation tasks.

Original: Following Geifman & El-Yaniv (2017; 2019)
Revised: Following Geifman & El-Yaniv (2017; 2019)

Original: ; Liu et al.
Revised: ; Liu et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2020), we use CIFAR-10, CIFAR-100 (Krizhevsky, 2009), and SVHN (Netzer et al., 2011) for image classiﬁcation tasks, and following Xin et al.
Revised: (2020), we use CIFAR-10, CIFAR-100 (Krizhevsky, 2009), and SVHN (Netzer et al., 2011) for image classiﬁcation tasks, and following Xin et al.

Original: (2021), we use MRPC (Dolan & Brockett, 2005), MNLI (Williams et al., 2018), and QNLI (Wang et al., 2018) for text classiﬁcation tasks.
Revised: (2021), we use MRPC (Dolan & Brockett, 2005), MNLI (Williams et al., 2018), and QNLI (Wang et al., 2018) for text classiﬁcation tasks.

Original: The usage of the training set and test set in selective classiﬁcation are the same as the standard classiﬁcation, because current selective classiﬁcation focuses on in-domain data (i.e., data from the same distribution as the training set)
Revised: The usage of the training set and test set in selective classiﬁcation are the same as the standard classiﬁcation, because current selective classiﬁcation focuses on in-domain data (i.e., data from the same distribution as the training set)

Original: (Geifman & El-Yaniv, 2017).
Revised: (Geifman & El-Yaniv, 2017).

Original: For example, if the selective classiﬁer is trained on the training set of CIFAR-10 (Krizhevsky, 2009), then it will be tested on the test set of CIFAR-10.
Revised: For example, if the selective classiﬁer is trained on the training set of CIFAR-10 (Krizhevsky, 2009), then it will be tested on the test set of CIFAR-10.

Original: Furthermore, MNLI’s development set and test set are divided into matched and mismatched parts.
Revised: Furthermore, MNLI’s development set and test set are divided into matched and mismatched parts.

Original: The matched parts are sampled from the same source as the training set (so they are in-domain data), while the mismatched parts are sampled from different sources.
Revised: The matched parts are sampled from the same source as the training set (so they are in-domain data), while the mismatched parts are sampled from different sources.

Original: In our experiments, only the matched parts are used since the current selective classiﬁcation only considers in-domain data.
Revised: In our experiments, only the matched parts are used since the current selective classiﬁcation only considers in-domain data.

Original: In addition, test sets of MRPC, QNLI, and MNLI are not accessible, so we use their development sets as test sets.
Revised: In addition, test sets of MRPC, QNLI, and MNLI are not accessible, so we use their development sets as test sets.

Original: Following Liu et al.
Revised: Following Liu et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2020), since CIFAR-10, CIFAR-100 and
Revised: (2020), since CIFAR-10, CIFAR-100 and

Original: SVHN originally had no development set, their development sets were 2000 samples randomly split from corresponding test sets.
Revised: SVHN originally had no development set, their development sets were 2000 samples randomly split from corresponding test sets.

Original: More details of all the datasets in our experiments are described in
Revised: More details of all the datasets in our experiments are described in

Original: Appendix C.1.
Revised: Appendix C.1.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: The evaluation metrics are AURC and selective risk (the lower, the better for both).
Revised: The evaluation metrics are AURC and selective risk (the lower, the better for both).

Original: AURC is a comprehensive metric of selective classiﬁers, and selective risk is a standard metric in previous works (Geifman & El-Yaniv, 2019; Liu et al., 2019; Huang et al., 2020).
Revised: AURC is a comprehensive metric of selective classiﬁers, and selective risk is a standard metric in previous works (Geifman & El-Yaniv, 2019; Liu et al., 2019; Huang et al., 2020).

Original: In this paper, given a selective classiﬁer, the result of selective risk is shown in the form of risk-coverage curves, which shows the selective risk of the selective classiﬁer against its coverage.
Revised: In this paper, given a selective classiﬁer, the result of selective risk is shown in the form of risk-coverage curves, which shows the selective risk of the selective classiﬁer against its coverage.

Original: According to the object of selective classiﬁcation, a selective classiﬁer with a lower risk-coverage curve is better.
Revised: According to the object of selective classiﬁcation, a selective classiﬁer with a lower risk-coverage curve is better.

Original: Networks.
Revised: Networks.

Original: Following Huang et al.
Revised: Following Huang et al.

Original: ; Xin et al.
Revised: ; Xin et al.

Original: (2021), for image classiﬁcation and text classiﬁcation, we use VGG-16 (Simonyan & Zisserman, 2014) and BERT-base (Devlin et al., 2019) as the backbones of selective classiﬁers, respectively.
Revised: (2021), for image classiﬁcation and text classiﬁcation, we use VGG-16 (Simonyan & Zisserman, 2014) and BERT-base (Devlin et al., 2019) as the backbones of selective classiﬁers, respectively.

Original: More details of the backbone models and their training procedures are provided in Appendix C.2.
Revised: More details of the backbone models and their training procedures are provided in Appendix C.2.

Original: Baselines.
Revised: Baselines.

Original: We use SR (Geifman & El-Yaniv, 2017), SN (SelectiveNet) (Geifman & El-Yaniv, 2019),
Revised: We use SR (Geifman & El-Yaniv, 2017), SN (SelectiveNet) (Geifman & El-Yaniv, 2019),

Original: Gambler (Liu et al., 2019), SAT (Huang et al., 2020), and Reg-curr (Xin et al., 2021) for both image classiﬁcation and text classiﬁcation.
Revised: Gambler (Liu et al., 2019), SAT (Huang et al., 2020), and Reg-curr (Xin et al., 2021) for both image classiﬁcation and text classiﬁcation.

Original: Note that the SN is optimized for ﬁxed coverage, so the comprehensive metrics AURC and risk-coverage curve, which summarizes performances under different coverages, are not suitable for evaluating the SN.
Revised: Note that the SN is optimized for ﬁxed coverage, so the comprehensive metrics AURC and risk-coverage curve, which summarizes performances under different coverages, are not suitable for evaluating the SN.

Original: Thus, we only evaluate the selective risk for a ﬁxed coverage of the SN ensemble and provide the results in Appendix E.
Revised: Thus, we only evaluate the selective risk for a ﬁxed coverage of the SN ensemble and provide the results in Appendix E.

Original: To ensure a fair comparison, for tasks of image classiﬁcation, all the baselines are re-implemented based on the open resource code of SAT (Huang et al., 2020), and for tasks of text classiﬁcation, they are re-implemented based on the open resource code of Reg-curr (Xin et al., 2021).
Revised: To ensure a fair comparison, for tasks of image classiﬁcation, all the baselines are re-implemented based on the open resource code of SAT (Huang et al., 2020), and for tasks of text classiﬁcation, they are re-implemented based on the open resource code of Reg-curr (Xin et al., 2021).

Original: The details of the hyperparameters of each baseline are provided in Appendix C.3.
Revised: The details of the hyperparameters of each baseline are provided in Appendix C.3.

Original: 6. V ERIFICATION OF THE A SSUMPTIONS
Revised: 6. V ERIFICATION OF THE A SSUMPTIONS

Original: We examine Assumption 1 and Assumption 2 ( only for the baseline of SR) on datasets for both image classiﬁcation and text classiﬁcation.
Revised: We examine Assumption 1 and Assumption 2 ( only for the baseline of SR) on datasets for both image classiﬁcation and text classiﬁcation.

Original: Since the deﬁnite samples are the idealization of samples on which member models provide almost the same predictive probability distributions, we take samples with a standard deviation of predictive probability distributions of member models (or STD for short) less than a small positive number (cid:15) as deﬁnite samples and the other samples as ambiguous samples in experiments.
Revised: Since the deﬁnite samples are the idealization of samples on which member models provide almost the same predictive probability distributions, we take samples with a standard deviation of predictive probability distributions of member models (or STD for short) less than a small positive number (cid:15) as deﬁnite samples and the other samples as ambiguous samples in experiments.

Original: Formally, STD := (cid:113)
Revised: Formally, STD := (cid:113)

Original: [Equation]
Revised: [Equation]

Original: , where ˆ π j is the predictive probability distribution vector of the j -th member model, and samples with STD < (cid:15) approximates deﬁnite samples in experiments.
Revised: , where ˆ π j is the predictive probability distribution vector of the j -th member model, and samples with STD < (cid:15) approximates deﬁnite samples in experiments.

Original: We choose (cid:15) =
Revised: We choose (cid:15) =

Original: 10 − 3 for datasets of image classiﬁcation and (cid:15) =
Revised: 10 − 3 for datasets of image classiﬁcation and (cid:15) =

Original: 10 − 2 for datasets of text classiﬁcation.
Revised: 10 − 2 for datasets of text classiﬁcation.

Original: Figure 1(a) and Figure 1(e) show the selective error rates (selective risks) of samples with STD < 10 − 3 and samples with STD ≥ 10 − 3 given a range of conﬁdence thresholds (which approximate Pr( Err | D, C ≥ τ ) and Pr( Err | A, C ≥ τ ) ) on the test set of each dataset.
Revised: Figure 1(a) and Figure 1(e) show the selective error rates (selective risks) of samples with STD < 10 − 3 and samples with STD ≥ 10 − 3 given a range of conﬁdence thresholds (which approximate Pr( Err | D, C ≥ τ ) and Pr( Err | A, C ≥ τ ) ) on the test set of each dataset.

Original: The results show that the selective risk of samples with STD < 10 − 3 is lower than that of samples with STD ≥ 10 − 3 for all conﬁdence thresholds near 1 on all datasets, which veriﬁes
Revised: The results show that the selective risk of samples with STD < 10 − 3 is lower than that of samples with STD ≥ 10 − 3 for all conﬁdence thresholds near 1 on all datasets, which veriﬁes

Original: Assumption 1.
Revised: Assumption 1.

Original: Figure 1(b)-1(d) and Figure 1(f)-1(h) shows the histogram of conﬁdence scores of samples with STD < (cid:15) and that of other samples, which approximate p C ( κ | D ) and p C ( κ | A ) , on the test set of each dataset.
Revised: Figure 1(b)-1(d) and Figure 1(f)-1(h) shows the histogram of conﬁdence scores of samples with STD < (cid:15) and that of other samples, which approximate p C ( κ | D ) and p C ( κ | A ) , on the test set of each dataset.

Original: The results show that the number of samples with STD < 10 − 3 is non-zero in the top bin on all datasets, which veriﬁes Assumption 2.
Revised: The results show that the number of samples with STD < 10 − 3 is non-zero in the top bin on all datasets, which veriﬁes Assumption 2.

Original: In summary, Assumption 1 and 2 hold on all datasets.
Revised: In summary, Assumption 1 and 2 hold on all datasets.

Original: [Figure]
Revised: [Figure]

Original: 6. E VALUATION OF S ELECTIVE C LASSIFIER E NSEMBLES
Revised: 6. E VALUATION OF S ELECTIVE C LASSIFIER E NSEMBLES

Original: We ﬁrst verify Theorem 2.
Revised: We ﬁrst verify Theorem 2.

Original: Figure 2 shows the risk-coverage curves of the ensembles and the individual models of each baseline on each dataset.
Revised: Figure 2 shows the risk-coverage curves of the ensembles and the individual models of each baseline on each dataset.

Original: As we can see, except on MRPC, the risk-coverage curve of the ensemble is always entirely below that of the individual model, i.e., the ensemble has a lower selective risk than the individual model under any coverage, which is consistent with Theorem 2.
Revised: As we can see, except on MRPC, the risk-coverage curve of the ensemble is always entirely below that of the individual model, i.e., the ensemble has a lower selective risk than the individual model under any coverage, which is consistent with Theorem 2.

Original: The abnormal results on MRPC dataset may be because of the small number of samples in MRPC.
Revised: The abnormal results on MRPC dataset may be because of the small number of samples in MRPC.

Original: The development set of MRPC has only 0.4k samples, which is much smaller than development sets of other datasets (see Table 2).
Revised: The development set of MRPC has only 0.4k samples, which is much smaller than development sets of other datasets (see Table 2).

Original: More importantly, when the coverage is low, say 10%, only about 40 samples in MRPC are selected to predict, which may cause a large variance in selective risk estimation.
Revised: More importantly, when the coverage is low, say 10%, only about 40 samples in MRPC are selected to predict, which may cause a large variance in selective risk estimation.

Original: Thus, the estimation of selective risk is not accurate under low coverage, which may explain the violation of Theorem 2 on MRPC.
Revised: Thus, the estimation of selective risk is not accurate under low coverage, which may explain the violation of Theorem 2 on MRPC.

Original: In summary, except the results on MRPC, which might have a large variance in selective risk estimation, the experimental results in Figureverify the correctness and practicability of Theorem 2.
Revised: In summary, except the results on MRPC, which might have a large variance in selective risk estimation, the experimental results in Figureverify the correctness and practicability of Theorem 2.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Next, we show the AURCs of the ensembles in Table 1, where o is the hyperparameter of Deep
Revised: Next, we show the AURCs of the ensembles in Table 1, where o is the hyperparameter of Deep

Original: Gambler, and each ensemble has 2-5 member models.
Revised: Gambler, and each ensemble has 2-5 member models.

Original: As we can see, the ensemble with ﬁve members has a signiﬁcantly lower AURC than the individual model on each dataset.
Revised: As we can see, the ensemble with ﬁve members has a signiﬁcantly lower AURC than the individual model on each dataset.

Original: On CIFAR-10,
Revised: On CIFAR-10,

Original: CIFAR-100, SVHN, MRPC, QNLI, and MNLI, the AURCs of the best ensemble models with ﬁve members are 23%, 15%, 7%, 14%, 14%, and 9% lower than those of the best individual models.
Revised: CIFAR-100, SVHN, MRPC, QNLI, and MNLI, the AURCs of the best ensemble models with ﬁve members are 23%, 15%, 7%, 14%, 14%, and 9% lower than those of the best individual models.

Original: Among all ensembles, SR ensemble has the lowest AURCs on CIFAR-10, CIFAR-100, and MRPC,
Revised: Among all ensembles, SR ensemble has the lowest AURCs on CIFAR-10, CIFAR-100, and MRPC,

Original: Surprisingly, the SR ensemble and Reg-curr ensemble are state-of-the-art selective classiﬁers for image and text classiﬁcation tasks, respectively.
Revised: Surprisingly, the SR ensemble and Reg-curr ensemble are state-of-the-art selective classiﬁers for image and text classiﬁcation tasks, respectively.

Original: They only use the maximum probability as the conﬁdence score, while SAT and SN use more sophisticated and explicit conﬁdence score functions.
Revised: They only use the maximum probability as the conﬁdence score, while SAT and SN use more sophisticated and explicit conﬁdence score functions.

Original: Note that, on SVHN, the SAT ensemble performs better than the SR ensemble.
Revised: Note that, on SVHN, the SAT ensemble performs better than the SR ensemble.

Original: We ﬁnd that the annotations on SVHN are noisy, and after manually removing some noisy samples in the training set, the SR ensemble performs better than SAT ensemble.
Revised: We ﬁnd that the annotations on SVHN are noisy, and after manually removing some noisy samples in the training set, the SR ensemble performs better than SAT ensemble.

Original: Therefore, the SVHN result indicates that
Revised: Therefore, the SVHN result indicates that

Original: SAT is better at handling label noise rather than selecting when to abstain.
Revised: SAT is better at handling label noise rather than selecting when to abstain.

Original: Please refer to Appendix
Revised: Please refer to Appendix

Original: F.3 for more details.
Revised: F.3 for more details.

Original: In addition, we conduct experiments to explore further properties of the selective classiﬁer ensemble, including the effect of the number of members and the relationship between the classiﬁcation performance and the selective classiﬁcation performance (see Appendix F).
Revised: In addition, we conduct experiments to explore further properties of the selective classiﬁer ensemble, including the effect of the number of members and the relationship between the classiﬁcation performance and the selective classiﬁcation performance (see Appendix F).

Original: The results are that an ensemble with more member models has better selective performance, and good classiﬁcation performance of the ensemble does not necessarily imply good selective classiﬁcation performance.
Revised: The results are that an ensemble with more member models has better selective performance, and good classiﬁcation performance of the ensemble does not necessarily imply good selective classiﬁcation performance.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: A possible direction for future work is to adapt our analysis to standard classiﬁcation .
Revised: A possible direction for future work is to adapt our analysis to standard classiﬁcation .

Original: The previous analyses of the randomization-based ensemble for standard classiﬁcation need some impractical assumptions (Tumer & Ghosh, 1996; Fumera & Roli, 2005).
Revised: The previous analyses of the randomization-based ensemble for standard classiﬁcation need some impractical assumptions (Tumer & Ghosh, 1996; Fumera & Roli, 2005).

Original: On the contrary, this paper’s assumptions are a good approximation of practical settings (see Section 6.1), and more importantly, the standard classiﬁcation is a particular case of selective classiﬁcation, i.e., selective classiﬁcation with coverage of 1.
Revised: On the contrary, this paper’s assumptions are a good approximation of practical settings (see Section 6.1), and more importantly, the standard classiﬁcation is a particular case of selective classiﬁcation, i.e., selective classiﬁcation with coverage of 1.

Original: Therefore, our analysis (although it does not cover the case of coverage of 1, i.e., the case of standard classiﬁcation) may motivate the analysis of the randomization-based ensemble for standard classiﬁcation in practical settings.
Revised: Therefore, our analysis (although it does not cover the case of coverage of 1, i.e., the case of standard classiﬁcation) may motivate the analysis of the randomization-based ensemble for standard classiﬁcation in practical settings.

Original: Another possible direction for future work is the relaxation of assumptions.
Revised: Another possible direction for future work is the relaxation of assumptions.

Original: This paper’s assumption is a little strong for the convenience of proof.
Revised: This paper’s assumption is a little strong for the convenience of proof.

Original: Although the experimental results suggest that these assumptions are the actual behaviors of the SR model, we guess the assumptions can be relaxed while the conclusion keeps the same.
Revised: Although the experimental results suggest that these assumptions are the actual behaviors of the SR model, we guess the assumptions can be relaxed while the conclusion keeps the same.

Original: It is interesting to relax the assumption of the idealization of the deﬁnite samples.
Revised: It is interesting to relax the assumption of the idealization of the deﬁnite samples.

Original: Although the idealization may be a good approximation of practical setting, it is unrealistic anyway.
Revised: Although the idealization may be a good approximation of practical setting, it is unrealistic anyway.

Original: We believe that a similar theoretical result holds in the absence of the idealization.
Revised: We believe that a similar theoretical result holds in the absence of the idealization.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We prove that under some assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.
Revised: We prove that under some assumptions, the ensemble has a lower selective risk than the individual model under a range of coverage.

Original: Although the metrics of selective classiﬁcation are non-convex, we complete the proof with the help of several assumptions motivated by empirical observations.
Revised: Although the metrics of selective classiﬁcation are non-convex, we complete the proof with the help of several assumptions motivated by empirical observations.

Original: The assumptions and the result are well supported by the experimental results on multiple datasets of image classiﬁcation tasks and text classiﬁcation tasks.
Revised: The assumptions and the result are well supported by the experimental results on multiple datasets of image classiﬁcation tasks and text classiﬁcation tasks.

Original: A surprising empirical result is that two simple methods, SR ensemble and its variant Reg-curr ensemble, (which can be summarized as the ensemble models with maximum probability as conﬁdence ) are state-of-the-art selective classiﬁers.
Revised: A surprising empirical result is that two simple methods, SR ensemble and its variant Reg-curr ensemble, (which can be summarized as the ensemble models with maximum probability as conﬁdence ) are state-of-the-art selective classiﬁers.

Original: A R ELATIONSHIP BETWEEN SELECTIVE CLASSIFICATION AND OOD DETECTION
Revised: A R ELATIONSHIP BETWEEN SELECTIVE CLASSIFICATION AND OOD DETECTION

Original: Figure 3 shows the relationship between selective classiﬁcation and OOD detection (or open set recognition, novelty detection).
Revised: Figure 3 shows the relationship between selective classiﬁcation and OOD detection (or open set recognition, novelty detection).

Original: B P ROOFS
Revised: B P ROOFS

Original: The complete proof is somewhat complex but its intuition is straightforward.
Revised: The complete proof is somewhat complex, but its intuition is straightforward.

Original: In a word, the intuition is that because the ensemble avoids to be overconﬁdent over ambiguous samples, the ensemble has a lower selective risk.
Revised: In a word, the intuition is that because the ensemble avoids being overconﬁdent over ambiguous samples, the ensemble has a lower selective risk.

Original: Details of the intuition are as follows:
Revised: Details of the intuition are as follows:

Original: the individual model is not ”modest” over both ambiguous samples and deﬁnite samples (Assumption 2);
Revised: the individual model is not ”modest” over both ambiguous samples and deﬁnite samples (Assumption 2);

Original: by contrast, based on Assumption 3, we prove that the ensemble provides modest conﬁdence to ambiguous samples (Proposition 1).
Revised: by contrast, based on Assumption 3, we prove that the ensemble provides modest conﬁdence to ambiguous samples (Proposition 1).

Original: In addition, the conﬁdence of deﬁnite samples remains the same throughout ensembling (due to the deﬁnition of deﬁnite samples); 3.
Revised: In addition, the conﬁdence of deﬁnite samples remains the same throughout ensembling (due to the deﬁnition of deﬁnite samples); 3.

Original: thus, when conﬁdence approaches 1, as long as thethe classiﬁer’s error rate over deﬁnite samples is lower than the error rate over ambiguous samples (Assumption 1), the individual model suffers more error predictions that come with the ambiguous samples than the ensemble.
Revised: thus, when conﬁdence approaches 1, as long as the classiﬁer’s error rate over deﬁnite samples is lower than the error rate over ambiguous samples (Assumption 1), the individual model suffers more error predictions that come with the ambiguous samples than the ensemble.

Original: Based on this, we prove that the selective risk drops under a range of coverage via ensembling (Theorem 2).
Revised: Based on this, we prove that the selective risk drops under a range of coverage via ensembling (Theorem 2).

Original: We prove Proposition 1 in Section B.1 and prove Theorem 2 in Section B.2.
Revised: We prove Proposition 1 in Section B.1 and prove Theorem 2 in Section B.2.

Original: The road map of the complete proof of the ﬁnal result, i.e., Theorem 2, is shown in Figure 4.
Revised: The road map of the complete proof of the ﬁnal result, i.e., Theorem 2, is shown in Figure 4.

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: To prove Proposition 1, that is, the ensemble is modest over ambiguous samples, we ﬁrst show that over ambiguous samples, the ensemble provides a moderate predictive probability for each class.
Revised: To prove Proposition 1, that is, the ensemble is modest over ambiguous samples, we ﬁrst show that over ambiguous samples, the ensemble provides a moderate predictive probability for each class.

Original: By moderate predictive probability , we mean a predictive probability whose PDF approaches zero when itself approaches one or zero 4 .
Revised: By moderate predictive probability , we mean a predictive probability whose PDF approaches zero when itself approaches one or zero 4 .

Original: Formally, we have the following lemma (proof is provided in
Revised: Formally, we have the following lemma (proof is provided in

Original: Section B.1.1).
Revised: Section B.1.1).

Original: Lemma 3.
Revised: Lemma 3.

Original: If Assumption 3 holds, then
Revised: If Assumption 3 holds, then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (11) and
Revised: (11) and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (13) where the notation follows that of Assumption 3, and Π k ens is the predictive probability of the ensemble for class k .
Revised: (13) where the notation follows that of Assumption 3, and Π k ens is the predictive probability of the ensemble for class k .

Original: Secondly, to prove the ensemble is modest over ambiguous samples, we show the relationship between the PDF of conﬁdence and the PDFs of predictive probabilities.
Revised: Secondly, to prove the ensemble is modest over ambiguous samples, we show the relationship between the PDF of conﬁdence and the PDFs of predictive probabilities.

Original: Note that the conﬁdence of an SR model is the maximum predictive probabilities.
Revised: Note that the conﬁdence of an SR model is the maximum predictive probability.

Original: Thus, the following lemma (proof is provided in Section B.1.2) bounds the PDF of conﬁdence by PDFs of predictive probabilities, and then it is clear that the ensemble is modest over ambiguous samples, considering the moderate predictive probabilities of the ensemble over ambiguous samples.
Revised: Thus, the following lemma (proof is provided in Section B.1.2) bounds the PDF of conﬁdence by PDFs of predictive probabilities, and then it is clear that the ensemble is modest over ambiguous samples, considering the moderate predictive probabilities of the ensemble over ambiguous samples.

Original: Lemma 4.
Revised: Lemma 4.

Original: Let Π k (1 ≤ k ≤ K ) be K continuous random variables, and C :=
Revised: Let Π k (1 ≤ k ≤ K ) be K continuous random variables, and C :=

Original: max k Π k .
Revised: max k Π k .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Finally, since C ens = max k Π k ens , Lemma 3 and Lemma 4 derive that when κ ens → 1 − ,
Revised: Finally, since C ens = max k Π k ens , Lemma 3 and Lemma 4 derive that when κ ens → 1 − ,

Original: [Equation]
Revised: [Equation]

Original: and thus Proposition 1 holds.
Revised: and thus Proposition 1 holds.

Original: B.1. P ROOF OF L EMMA
Revised: B.1. P ROOF OF L EMMA

Original: We ﬁrst derive the PDF of the average of multiple continuous random variables in terms of the
Revised: We ﬁrst derive the PDF of the average of multiple continuous random variables in terms of the

Original: PDFs of these random variables (Lemma 5), which helps us to analyze the PDF of the ensemble’s predictive probabilities.
Revised: PDFs of these random variables (Lemma 5), which helps us to analyze the PDF of the ensemble’s predictive probabilities.

Original: Lemma 5.
Revised: Lemma 5.

Original: Let X 1 , X 2 , . . .
Revised: Let X 1 , X 2 , . . .

Original: X M be M continuous random variables, and their average is X avg := 1 M (cid:80) Mi
Revised: X M be M continuous random variables, and their average is X avg := 1 M (cid:80) Mi

Original: =1 X i .
Revised: =1 X i .

Original: Then the PDF of X avg is
Revised: Then the PDF of X avg is

Original: [Equation]
Revised: [Equation]

Original: where p (cid:126) X is p X 1 , X 2 ,..., X M for short.
Revised: where p (cid:126) X is p X 1 , X 2 ,..., X M for short.

Original: Proof.
Revised: Proof.

Original: The distribution function of X avg is
Revised: The distribution function of X avg is

Original: [Equation]
Revised: [Equation]

Original: Let x M = u − (cid:80) M − 1 i =1 x i , then the integral above is equal to
Revised: Let x M = u − (cid:80) M − 1 i =1 x i , then the integral above is equal to

Original: [Equation]
Revised: [Equation]

Original: The PDF of X avg is the derivative of F X avg , which, combined with (,16) derives
Revised: The PDF of X avg is the derivative of F X avg , which, combined with (,16) derives

Original: [Equation]
Revised: [Equation]

Original: which is exactly (15)
Revised: which is exactly (15)

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 3.
Revised: Proof of Lemma 3.

Original: Based on Lemma 5 and Assumption 3, we prove Lemma 3 as follows.
Revised: Based on Lemma 5 and Assumption 3, we prove Lemma 3 as follows.

Original: Proof.
Revised: Proof.

Original: With Lemma 5 applied to Π ki , 1 ≤ i ≤ M , we have
Revised: With Lemma 5 applied to Π ki , 1 ≤ i ≤ M , we have

Original: [Equation]
Revised: [Equation]

Original: where p (cid:126) Π k is p Π k 1 ,..., Π kM for short.
Revised: where p (cid:126) Π k is p Π k 1 ,..., Π kM for short.

Original: The integrand in the right-hand side of (17) being non-zero requires
Revised: The integrand in the right-hand side of (17) being non-zero requires

Original: [Equation]
Revised: [Equation]

Original: Firstly, we prove (10) and (12).
Revised: Firstly, we prove (10) and (12).

Original: When 0 ≤ π k ens ≤ 1 M , it easy to verify that (18) is equivalent to
Revised: When 0 ≤ π k ens ≤ 1 M , it easy to verify that (18) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Thus, (17) transforms into p
Revised: Thus, (17) transforms into p

Original: [Equation]
Revised: [Equation]

Original: Considering that p (cid:126) Π k ( ·| A ) is bounded, as Assumption 3 claims, let B be one of its upper bounds.
Revised: Considering that p (cid:126) Π k ( ·| A ) is bounded, as Assumption 3 claims, let B be one of its upper bounds.

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: which directly derives (10) and (12) (note that the PDF is non-negative).
Revised: which directly derives (10) and (12) (note that the PDF is non-negative).

Original: Secondly, we prove (11) and (13).
Revised: Secondly, we prove (11) and (13).

Original: These two equations could be derived like (10) and (12).
Revised: These two equations could be derived like (10) and (12).

Original: However, here we take another way that uses a little trick to simplify the proof.
Revised: However, here we take another way that uses a little trick to simplify the proof.

Original: Let Π ki and Π k ens be the corresponding random variables of π ki and π k ens respectively, and U i =
Revised: Let Π ki and Π k ens be the corresponding random variables of π ki and π k ens respectively, and U i =

Original: 1 − Π ki ,
Revised: 1 − Π ki ,

Original: U ens = 1 M (cid:80) M
Revised: U ens = 1 M (cid:80) M

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: U i =
Revised: U i =

Original: 1 − Π k ens .
Revised: 1 − Π k ens .

Original: Applying (20) to U i and U ens , we get that when 0 ≤ u ens ≤ 1 M ,
Revised: Applying (20) to U i and U ens , we get that when 0 ≤ u ens ≤ 1 M ,

Original: [Equation]
Revised: [Equation]

Original: It is easy to see that p U ens ( u ens | A ) =
Revised: It is easy to see that p U ens ( u ens | A ) =

Original: p Π k ens (1 − u ens | A ) .
Revised: p Π k ens (1 − u ens | A ) .

Original: Sticking this into (21), we have
Revised: Sticking this into (21), we have

Original: [Equation]
Revised: [Equation]

Original: when 0 ≤ u ens ≤ 1 M .
Revised: when 0 ≤ u ens ≤ 1 M .

Original: With the (1 − u ens ) in the equation above replaced with π k ens , we have
Revised: With the (1 − u ens ) in the equation above replaced with π k ens , we have

Original: [Equation]
Revised: [Equation]

Original: when 1 − 1 M ≤ π k ens ≤ 1 , which directly derives (11) and (13).
Revised: when 1 − 1 M ≤ π k ens ≤ 1 , which directly derives (11) and (13).

Original: B.1. P ROOF OF L EMMA
Revised: B.1. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: First of all, we prove ∀ κ 1 , κ 2 , κ 1 < κ 2 ,
Revised: First of all, we prove ∀ κ 1 , κ 2 , κ 1 < κ 2 ,

Original: [Equation]
Revised: [Equation]

Original: It is easy to see that
Revised: It is easy to see that

Original: [Equation]
Revised: [Equation]

Original: so the left-hand side of (22) is (cid:90)
Revised: so the left-hand side of (22) is (cid:90)

Original: [Equation]
Revised: [Equation]

Original: where the last equality is due to ( −∞ , κ 1 ] ⊂ ( −∞ , κ 2 ] , and the right-hand side of (22) is
Revised: where the last equality is due to ( −∞ , κ 1 ] ⊂ ( −∞ , κ 2 ] , and the right-hand side of (22) is

Original: K (cid:88)
Revised: K (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because R k − 1 × [ κ 1 , κ 2 ] × R K − k for different k , 1 ≤ k ≤ K , may have an intersection.
Revised: where the last inequality is because R k − 1 × [ κ 1 , κ 2 ] × R K − k for different k , 1 ≤ k ≤ K , may have an intersection.

Original: To prove (22), we only need to prove that the right-hand side of (23) is less than or equal to the right-hand side of (24), which is equivalent to prove
Revised: To prove (22), we only need to prove that the right-hand side of (23) is less than or equal to the right-hand side of (24), which is equivalent to prove

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now we prove (25).
Revised: Now we prove (25).

Original: ∀ ( π 1 , . . .
Revised: ∀ ( π 1 , . . .

Original: , π K ) ∈ ( −∞ , κ 2 ] K \ ( −∞ , κ 1 ] K , we have
Revised: , π K ) ∈ ( −∞ , κ 2 ] K \ ( −∞ , κ 1 ] K , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 27) where (27) is because if all π k is less than or equal to κ 1 instead, then ( π 1 , . .
Revised: 27) where (27) is because if all π k is less than or equal to κ 1 instead, then ( π 1 , . .

Original: . , π K ) ∈ ( −∞ , κ 1 ] K , which contradicts with ( π 1 , . .
Revised: . , π K ) ∈ ( −∞ , κ 1 ] K , which contradicts with ( π 1 , . .

Original: . , π K ) ∈ ( −∞ , κ 2 ] K \ ( −∞ , κ 1 ] K .
Revised: . , π K ) ∈ ( −∞ , κ 2 ] K \ ( −∞ , κ 1 ] K .

Original: Thus, π k 0 ∈ [ κ 1 , κ 2 ] , so ( π 1 , . . .
Revised: Thus, π k 0 ∈ [ κ 1 , κ 2 ] , so ( π 1 , . . .

Original: , π K ) ∈ R k 0 − 1 ×
Revised: , π K ) ∈ R k 0 − 1 ×

Original: [ κ 1 , κ 2 ] × R K − k 0 ⊂
Revised: [ κ 1 , κ 2 ] × R K − k 0 ⊂

Original: K (cid:91) k
Revised: K (cid:91) k

Original: =
Revised: =

Original: R k − 1 × [ κ 1 , κ 2 ] × R K − k , which is precisely (25), and therefore (22) is proved.
Revised: R k − 1 × [ κ 1 , κ 2 ] × R K − k , which is precisely (25), and therefore (22) is proved.

Original: With (22) and the deﬁnition of derivatives, it is easy to see that F (cid:48) C ( κ ) ≤ (cid:80) K k =1 F (cid:48)
Revised: With (22) and the deﬁnition of derivatives, it is easy to see that F (cid:48) C ( κ ) ≤ (cid:80) K k =1 F (cid:48)

Original: Π k ( κ ) , which is equivalent to p C ( κ ) ≤ (cid:80)
Revised: Π k ( κ ) , which is equivalent to p C ( κ ) ≤ (cid:80)

Original: Kk =1 p Π k ( κ ) .
Revised: Kk =1 p Π k ( κ ) .

Original: Thus, Lemma 4 is proved.
Revised: Thus, Lemma 4 is proved.

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: The intuition of the proof is as follows.
Revised: The intuition of the proof is as follows.

Original: Intuitively, since the ensemble is more modest than the individual over ambiguous samples and is the same modest as the individual model over deﬁnite samples, the ensemble tends to select more deﬁnite samples when the conﬁdence threshold approaches 1, compared with the individual model (Lemma 7).
Revised: Intuitively, since the ensemble is more modest than the individual over ambiguous samples and is the same modest as the individual model over deﬁnite samples, the ensemble tends to select more deﬁnite samples when the conﬁdence threshold approaches 1, compared with the individual model (Lemma 7).

Original: Thus, (still intuitively) as long as the selective risk over deﬁnite samples is lower than that over ambiguous samples when the conﬁdence threshold approaches 1 (Assumption 1), the ensemble is certainly to have a lower selective risk when the conﬁdence threshold approaches 1.
Revised: Thus, (still intuitively) as long as the selective risk over deﬁnite samples is lower than that over ambiguous samples when the conﬁdence threshold approaches 1 (Assumption 1), the ensemble is certainly to have a lower selective risk when the conﬁdence threshold approaches 1.

Original: Although the intuition is straightforward, the rigorous proof is not easy.
Revised: Although the intuition is straightforward, the rigorous proof is not easy.

Original: For the convenience of the proof, we show Lemma 6 and Lemma 7 ﬁrst.
Revised: For the convenience of the proof, we show Lemma 6 and Lemma 7 ﬁrst.

Original: Lemma 6 claims that for the individual model, the selective risk given deﬁnite samples is lower than the overall selective risk.
Revised: Lemma 6 claims that for the individual model, the selective risk given deﬁnite samples is lower than the overall selective risk.

Original: Lemma 7 claims that the ensemble is unlikely to select ambiguous samples to predict when the conﬁdence threshold approaches 1.
Revised: Lemma 7 claims that the ensemble is unlikely to select ambiguous samples to predict when the conﬁdence threshold approaches 1.

Original: Lemma 6.
Revised: Lemma 6.

Original: If Assumption 1-2 holds, then for any individual SR model,
Revised: If Assumption 1-2 holds, then for any individual SR model,

Original: [Equation]
Revised: [Equation]

Original: where the notation follows those of Assumption 1-2.
Revised: where the notation follows those of Assumption 1-2.

Original: Proof.
Revised: Proof.

Original: Using Bayes’ rule
Revised: Using Bayes’ rule

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Using L’Hospital’s rule
Revised: Using L’Hospital’s rule

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to Assumption 2.
Revised: where the last inequality is due to Assumption 2.

Original: Combining this with (29)
Revised: Combining this with (29)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now we derive (28).
Revised: Now we derive (28).

Original: [Equation]
Revised: [Equation]

Original: According to the equation above, we have
Revised: According to the equation above, we have

Original: [Equation]
Revised: [Equation]

Original: Due to (31) and Assumption 1, the ﬁrst term of the equation above is positive, so (28) is derived.
Revised: Due to (31) and Assumption 1, the ﬁrst term of the equation above is positive, so (28) is derived.

Original: Lemma 7.
Revised: Lemma 7.

Original: If Assumption 2-3 hold, then
Revised: If Assumption 2-3 hold, then

Original: [Equation]
Revised: [Equation]

Original: where the notation follows those of Assumption 2-3.
Revised: where the notation follows those of Assumption 2-3.

Original: Proof.
Revised: Proof.

Original: Similar to (31), with Bayes’ rule and L’Hospital’s rule, we can derive that
Revised: Similar to (31), with Bayes’ rule and L’Hospital’s rule, we can derive that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where C ens is the conﬁdence score of the ensemble.
Revised: where C ens is the conﬁdence score of the ensemble.

Original: Because for a deﬁnite sample, the conﬁdence score of the ensemble is equal to that of the individual model, we have
Revised: Because for a deﬁnite sample, the conﬁdence score of the ensemble is equal to that of the individual model, we have

Original: [Equation]
Revised: [Equation]

Original: where the last equality is due to Proposition 1 and Assumption 2.
Revised: where the last equality is due to Proposition 1 and Assumption 2.

Original: Substituting this to (34), we obtain (33).
Revised: Substituting this to (34), we obtain (33).

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: Proof.
Revised: Proof.

Original: First, for the convenience of the proof, given an SR model ( f, g ) , we deﬁne a threshold-tocoverage function ρ ( f,g ) of ( f, g ) that maps the conﬁdence threshold to the corresponding coverage,
Revised: First, for the convenience of the proof, given an SR model ( f, g ) , we deﬁne a threshold-tocoverage function ρ ( f,g ) of ( f, g ) that maps the conﬁdence threshold to the corresponding coverage,

Original: [Equation]
Revised: [Equation]

Original: Second, we prove that ∃ δ ∈ (0 , 1) , ∀ τ ens ∈ (1 − δ, 1) ,
Revised: Second, we prove that ∃ δ ∈ (0 , 1) , ∀ τ ens ∈ (1 − δ, 1) ,

Original: [Equation]
Revised: [Equation]

Original: where C is the conﬁdence score of the individual SR model, Err ind is the event that the individual model makes an error prediction, Err ens is the event that the ensemble makes an error prediction, ρ and ρ ens are the threshold-to-coverage functions of the individual model and the ensemble respectively.
Revised: where C is the conﬁdence score of the individual SR model, Err ind is the event that the individual model makes an error prediction, Err ens is the event that the ensemble makes an error prediction, ρ and ρ ens are the threshold-to-coverage functions of the individual model and the ensemble respectively.

Original: Note that the symbol ρ − 1 denotes the preimage under ρ , rather than the inverse function of ρ .
Revised: Note that the symbol ρ − 1 denotes the preimage under ρ , rather than the inverse function of ρ .

Original: Because when τ ens → 1 − , the coverage of the ensemble ρ ens ( τ ens ) approaches 0, and the coverage of the individual model is equal to the coverage of the ensemble, i.e., ρ ( τ ) =
Revised: Because when τ ens → 1 − , the coverage of the ensemble ρ ens ( τ ens ) approaches 0, and the coverage of the individual model is equal to the coverage of the ensemble, i.e., ρ ( τ ) =

Original: ρ ens ( τ ens ) , we have τ → 1 − when τ ens → 1 − .
Revised: ρ ens ( τ ens ) , we have τ → 1 − when τ ens → 1 − .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: In addition, for deﬁnite samples, the conﬁdence score of the ensemble and that of the individual model are the same, and the ensemble and the individual model make error predictions on the same set of samples, i.e. Err ind =
Revised: In addition, for deﬁnite samples, the conﬁdence score of the ensemble and that of the individual model are the same, and the ensemble and the individual model make error predictions on the same set of samples, i.e. Err ind =

Original: Err ens , so
Revised: Err ens , so

Original: [Equation]
Revised: [Equation]

Original: where the second equality is just a variable substitution.
Revised: where the second equality is just a variable substitution.

Original: Finally, we have
Revised: Finally, we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality is due to Lemma 7, the second equality is due to (37) and (38)
Revised: where the ﬁrst equality is due to Lemma 7, the second equality is due to (37) and (38)

Original: [Equation]
Revised: [Equation]

Original: and the last inequality is due to Lemma 6.
Revised: and the last inequality is due to Lemma 6.

Original: Thus, with (39), it is easy to see that (35) holds.
Revised: Thus, with (39), it is easy to see that (35) holds.

Original: Third, we have
Revised: Third, we have

Original: [Equation]
Revised: [Equation]

Original: , D | C ens ≥ τ ens )
Revised: , D | C ens ≥ τ ens )

Original: =
Revised: =

Original: Pr( Err ens | A, C ens ≥
Revised: Pr( Err ens | A, C ens ≥

Original: τ ens )
Revised: τ ens )

Original: Pr( A | C ens ≥ τ ens )
Revised: Pr( A | C ens ≥ τ ens )

Original: + Pr( Err ens | D, C ens ≥
Revised: + Pr( Err ens | D, C ens ≥

Original: τ ens )
Revised: τ ens )

Original: Pr( D | C ens ≥ τ ens ) =
Revised: Pr( D | C ens ≥ τ ens ) =

Original: Pr( Err ens | A, C ens ≥
Revised: Pr( Err ens | A, C ens ≥

Original: τ ens )
Revised: τ ens )

Original: Pr( A | C ens ≥ τ ens )
Revised: Pr( A | C ens ≥ τ ens )

Original: + Pr( Err ens | D, C ens ≥ τ ens )
Revised: + Pr( Err ens | D, C ens ≥ τ ens )

Original: [1 − Pr( A | C ens ≥ τ ens )] =
Revised: [1 − Pr( A | C ens ≥ τ ens )] =

Original: [Pr( Err ens | A, C ens ≥ τ ens )
Revised: [Pr( Err ens | A, C ens ≥ τ ens )

Original: − Pr
Revised: − Pr

Original: ( Err ens | D, C ens ≥ τ ens )] · Pr( A | C ens ≥ τ ens ) + Pr( Err ens | D, C ens ≥ τ ens )
Revised: ( Err ens | D, C ens ≥ τ ens )] · Pr( A | C ens ≥ τ ens ) + Pr( Err ens | D, C ens ≥ τ ens )

Original: ≤ Pr( A | C ens ≥ τ ens )
Revised: ≤ Pr( A | C ens ≥ τ ens )

Original: + Pr( Err ens | D, C ens ≥ τ ens ) , (40) where the last inequality is because any probability is in [0, 1].
Revised: + Pr( Err ens | D, C ens ≥ τ ens ) , (40) where the last inequality is because any probability is in [0, 1].

Original: Combining this with (35), we have ∃ δ ∈ (0 , 1) , ∀ τ ens ∈ (
Revised: Combining this with (35), we have ∃ δ ∈ (0 , 1) , ∀ τ ens ∈ (

Original: [Equation]
Revised: [Equation]

Original: − δ, 1) ,
Revised: − δ, 1) ,

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is due to (35).
Revised: where the second inequality is due to (35).

Original: Pr( Err ens | C ens ≥ τ ens ) is the selective risk of the ensemble given the conﬁdence threshold of τ ens (or given coverage of ρ ens ( τ ens ) ), and Pr( Err ind | C ≥ τ ) is the selective risk of the individual model given the conﬁdence threshold of τ (or given coverage of ρ ( τ )
Revised: Pr( Err ens | C ens ≥ τ ens ) is the selective risk of the ensemble given the conﬁdence threshold of τ ens (or given coverage of ρ ens ( τ ens ) ), and Pr( Err ind | C ≥ τ ) is the selective risk of the individual model given the conﬁdence threshold of τ (or given coverage of ρ ( τ )

Original: =
Revised: =

Original: ρ ens ( τ ens ) ).
Revised: ρ ens ( τ ens ) ).

Original: These two selective risks are under the same coverage
Revised: These two selective risks are under the same coverage

Original: ρ ens ( τ ens ) .
Revised: ρ ens ( τ ens ) .

Original: Thus, (41) is equivalent to that ∃ δ ∈ (0 , 1) , ∀ τ ens ∈ (1 − δ, 1) , the ensemble has a lower selective risk than the individual model given the coverage of φ
Revised: Thus, (41) is equivalent to that ∃ δ ∈ (0 , 1) , ∀ τ ens ∈ (1 − δ, 1) , the ensemble has a lower selective risk than the individual model given the coverage of φ

Original: = ρ ens ( τ ens ) .
Revised: = ρ ens ( τ ens ) .

Original: This statement can be simpliﬁed as ∃ δ ∈ (0 , 1) , ∀ φ ∈ (0 ,
Revised: This statement can be simpliﬁed as ∃ δ ∈ (0 , 1) , ∀ φ ∈ (0 ,

Original: ρ ens (1 − δ )) , the ensemble has a lower selective risk than the individual model, given the coverage of φ .
Revised: ρ ens (1 − δ )) , the ensemble has a lower selective risk than the individual model, given the coverage of φ .

Original: C D ETAILS OF E XPERIMENTS C. D ATASETS
Revised: C D ETAILS OF E XPERIMENTS C. D ATASETS

Original: The experiments were conducted on multiple data sets of image classiﬁcation and text classiﬁcation.
Revised: The experiments were conducted on multiple data sets of image classiﬁcation and text classiﬁcation.

Original: The image classiﬁcation datasets are CIFAR-10, CIFAR-100, (Krizhevsky, 2009) and SVHN (Netzer et al., 2011), whose image sizes are all 32 × 32 × 3 pixels.
Revised: The image classiﬁcation datasets are CIFAR-10, CIFAR-100, (Krizhevsky, 2009) and SVHN (Netzer et al., 2011), whose image sizes are all 32 × 32 × 3 pixels.

Original: The datasets of text classiﬁcation are MRPC (Dolan & Brockett, 2005), MNLI (Williams et al., 2018) and QNLI (Wang et al., 2018).
Revised: The datasets of text classiﬁcation are MRPC (Dolan & Brockett, 2005), MNLI (Williams et al., 2018) and QNLI (Wang et al., 2018).

Original: The task of MRPC is to judge whether two paragraphs of text are semantically equivalent.
Revised: The task of MRPC is to judge whether two paragraphs of text are semantically equivalent.

Original: MNLI’s task is to judge the inferential relationship between sentences (three categories).
Revised: MNLI’s task is to judge the inferential relationship between sentences (three categories).

Original: The task of QNLI is to determine whether a paragraph has the answer to a given question.
Revised: The task of QNLI is to determine whether a paragraph has the answer to a given question.

Original: The sizes of the training set, development set, and test set of each data set used in experiments are shown in Table 2.
Revised: The sizes of the training set, development set, and test set of each data set used in experiments are shown in Table 2.

Original: MNLI’s development set and test set are divided into matched and mismatched parts.
Revised: MNLI’s development set and test set are divided into matched and mismatched parts.

Original: In the table, (m) represents matched, and (mm) represents mismatched.
Revised: In the table, (m) represents matched, and (mm) represents mismatched.

Original: The matched parts are sampled from the same source as the training set, while the mismatched parts are sampled from different sources.
Revised: The matched parts are sampled from the same source as the training set, while the mismatched parts are sampled from different sources.

Original: Current selective classiﬁcation only considers test samples from the same distribution as the training set, so only the matched parts are used in experiments.
Revised: Current selective classiﬁcation only considers test samples from the same distribution as the training set, so only the matched parts are used in experiments.

Original: In addition, test sets of MRPC, QNLI, and MNLI are not accessible, so we use their development sets as test sets.
Revised: In addition, test sets of MRPC, QNLI, and MNLI are not accessible, so we use their development sets as test sets.

Original: According to Liu et al.
Revised: According to Liu et al.

Original: Huang et al.
Revised: Huang et al.

Original: (2020), since CIFAR-10, CIFAR-100 and SVHN originally had no development set, their development sets were 2000 samples randomly divided from corresponding test sets.
Revised: (2020), since CIFAR-10, CIFAR-100 and SVHN originally had no development set, their development sets were 2000 samples randomly divided from corresponding test sets.

Original: [Table]
Revised: [Table]

Original: I MPLEMENTATIONS AND T RAINING P ROCEDURES
Revised: I MPLEMENTATIONS AND T RAINING P ROCEDURES

Original: For image classiﬁcation, the backbone model is VGG-16 (Simonyan & Zisserman, 2014) with
Revised: For image classiﬁcation, the backbone model is VGG-16 (Simonyan & Zisserman, 2014) with

Original: Dropout (Srivastava et al., 2014), batch normalization (Ioffe & Szegedy, 2015).
Revised: Dropout (Srivastava et al., 2014), batch normalization (Ioffe & Szegedy, 2015).

Original: It is trained in the same way as Huang et al.
Revised: It is trained in the same way as Huang et al.

Original: The model is optimized using SGD with an initial learning rate of 0.1 (the learning rate decays by half in every 25 epochs), the momentum of 0.9, weight decay of 0.0005, batch size of 128, and a total training epoch of 300.
Revised: The model is optimized using SGD with an initial learning rate of 0.1 (the learning rate decays by half in every 25 epochs), the momentum of 0.9, weight decay of 0.0005, batch size of 128, and a total training epoch of 300.

Original: Data preprocessing includes data augmentation (random cropping and ﬂip) and normalization.
Revised: Data preprocessing includes data augmentation (random cropping and ﬂip) and normalization.

Original: The implementations of the backbone model and data preprocessing are based on the ofﬁcial open-sourced implementation of SAT to ensure a fair comparison.
Revised: The implementations of the backbone model and data preprocessing are based on the ofﬁcial open-sourced implementation of SAT to ensure a fair comparison.

Original: For text classiﬁcation, the backbone model of selective classiﬁers is BERT-base (Devlin et al., 2019).
Revised: For text classiﬁcation, the backbone model of selective classiﬁers is BERT-base (Devlin et al., 2019).

Original: Pretrained BERT-base is provided by the Huggingface Transformer Library (Wolf et al., 2020).
Revised: Pretrained BERT-base is provided by the Huggingface Transformer Library (Wolf et al., 2020).

Original: It is trained/ﬁne-tuned in the same way as Xin et al.
Revised: It is trained/ﬁne-tuned in the same way as Xin et al.

Original: (2021), except on dataset MRPC.
Revised: (2021), except on dataset MRPC.

Original: On QNLI and
Revised: On QNLI and

Original: MNLI, the model is trained/ﬁne-tuned using AdamW (Loshchilov & Hutter, 2017) for 3 epochs, with a learning rate of 2 × 10 − 5 , batch size of 32, and the maximum input sequence length of 128.
Revised: MNLI, the model is trained/ﬁne-tuned using AdamW (Loshchilov & Hutter, 2017) for 3 epochs, with a learning rate of 2 × 10 − 5 , batch size of 32, and the maximum input sequence length of 128.

Original: On MRPC, the model is trained/ﬁne-tuned for 10 epoch, with other settings the same as those on
Revised: On MRPC, the model is trained/ﬁne-tuned for 10 epoch, with other settings the same as those on

Original: QNLI and MNLI.
Revised: QNLI and MNLI.

Original: This unique setting of training epoch is due to the small number of samples in
Revised: This unique setting of training epoch is due to the small number of samples in

Original: MRPC, which makes the training require more epochs to reach convergence on MRPC.
Revised: MRPC, which makes the training require more epochs to reach convergence on MRPC.

Original: C. H YPERPARAMETERS OF S ELECTIVE C LASSIFIERS
Revised: C. H YPERPARAMETERS OF S ELECTIVE C LASSIFIERS

Original: For the hyperparameter c of SN, we choose c = 0 .
Revised: For the hyperparameter c of SN, we choose c = 0 .

Original: 9 for evaluating its selective risk, given the coverage of 90%.
Revised: 9 for evaluating its selective risk, given the coverage of 90%.

Original: The results of SN are reported in Appendix E.
Revised: The results of SN are reported in Appendix E.

Original: For the hyperparameter o of
Revised: For the hyperparameter o of

Original: Gambler, we tune o on validation sets in the same way as Liu et al.
Revised: Gambler, we tune o on validation sets in the same way as Liu et al.

Original: For the hyperparameter α of SAT, we set α = 0 .
Revised: For the hyperparameter α of SAT, we set α = 0 .

Original: 99 , the same as Huang et al.
Revised: 99 , the same as Huang et al.

Original: For the hyperparameter λ of Reg-curr, we set λ = 0 . 05 .
Revised: For the hyperparameter λ of Reg-curr, we set λ = 0 . 05 .

Original: D A DDITIONAL E XPERIMENTAL R ESULTS
Revised: D A DDITIONAL E XPERIMENTAL R ESULTS

Original: Table 3 and 4 shows the selective risks of ensembles under coverage 10%-100% on each dataset, where hyperparameters of Gambler are the same as those in Table 1.
Revised: Table 3 and 4 shows the selective risks of ensembles under coverage 10%-100% on each dataset, where hyperparameters of Gambler are the same as those in Table 1.

Original: Notably, no ensemble consistently outperforms others under all coverage on all datasets, so it is not easy to tell which ensemble is state-of-the-art in this regard.
Revised: Notably, no ensemble consistently outperforms others under all coverage on all datasets, so it is not easy to tell which ensemble is state-of-the-art in this regard.

Original: This phenomenon is because different ensembles have similar overall performance but adopt different trade-offs between coverage and selective risk.
Revised: This phenomenon is because different ensembles have similar overall performance but adopt different trade-offs between coverage and selective risk.

Original: In this case, we need a comprehensive metric, e.g., AURC, to identify the state-of-the-art (see Table 1).
Revised: In this case, we need a comprehensive metric, e.g., AURC, to identify the state-of-the-art (see Table 1).

Original: E E MPIRICAL RESULTS OF SN
Revised: E E MPIRICAL RESULTS OF SN

Original: With the hyperparameter c of 0.9, we report the selective risks given the coverage of 90% of SN ensembles and the individual SN in Figure 5.
Revised: With the hyperparameter c of 0.9, we report the selective risks given the coverage of 90% of SN ensembles and the individual SN in Figure 5.

Original: The coverage is set to 90% because the target coverage of the SN is c (Geifman & El-Yaniv, 2019)), and c is 0.9 in our experiments.
Revised: The coverage is set to 90% because the target coverage of the SN is c (Geifman & El-Yaniv, 2019)), and c is 0.9 in our experiments.

Original: The results show that each ensemble of SN has a lower selective risk than the individual SN.
Revised: The results show that each ensemble of SN has a lower selective risk than the individual SN.

Original: F F URTHER P ROPERTIES OF S ELECTIVE C LASSIFIER E NSEMBLE F. T HE E FFECT OF N UMBER OF M EMBERS ON S ELECTIVE C LASSIFIER E NSEMBLE
Revised: F F URTHER P ROPERTIES OF S ELECTIVE C LASSIFIER E NSEMBLE F. T HE E FFECT OF N UMBER OF M EMBERS ON S ELECTIVE C LASSIFIER E NSEMBLE

Original: We evaluate AURCs of the SR ensemble, Gambler ensemble, and SAT ensemble of different numbers of members on CIFAR10, and ﬁnd that an ensemble with more members has a better performance, but is less efﬁcient.
Revised: We evaluate AURCs of the SR ensemble, Gambler ensemble, and SAT ensemble of different numbers of members on CIFAR10, and ﬁnd that an ensemble with more members has a better performance, but is less efﬁcient.

Original: The results are shown in Figure 6.
Revised: The results are shown in Figure 6.

Original: In most cases, the AURC on the test set of CIFAR-10 decreases as the number of members in the ensemble increases.
Revised: In most cases, the AURC on the test set of CIFAR-10 decreases as the number of members in the ensemble increases.

Original: In addition, as the number of members in the ensemble grows, the effect of adding one member drops.
Revised: In addition, as the number of members in the ensemble grows, the effect of adding one member drops.

Original: On the one hand, the result shows that an ensemble with a small number of members has good selective classiﬁcation performance.
Revised: On the one hand, the result shows that an ensemble with a small number of members has good selective classiﬁcation performance.

Original: On the other hand, it indicates that when the number of member models is large, increasing the number of members to improve the performance of the selective classiﬁcation ensemble is inefﬁcient.
Revised: On the other hand, it indicates that when the number of member models is large, increasing the number of members to improve the performance of the selective classiﬁcation ensemble is inefﬁcient.

Original: F. G OOD C LASSIFICATION P ERFORMANCE D OES N OT I MPLY G OOD S ELECTIVE C LASSIFICATION P ERFORMANCE
Revised: F. G OOD C LASSIFICATION P ERFORMANCE D OES N OT I MPLY G OOD S ELECTIVE C LASSIFICATION P ERFORMANCE

Original: It is well known that the ensemble has better classiﬁcation performance than an individual model, but this does not guarantee a better selective classiﬁcation performance of the ensemble.
Revised: It is well known that the ensemble has better classiﬁcation performance than an individual model, but this does not guarantee a better selective classiﬁcation performance of the ensemble.

Original: To demonstrate this, we design an SR model with a big backbone, and show that it has as good classiﬁcation
Revised: To demonstrate this, we design an SR model with a big backbone, and show that it has as good classiﬁcation

Original: [Table]
Revised: [Table]

Original: performance as an SR ensemble with a standard backbone but worse selective classiﬁcation performance than an SR model with a standard backbone.
Revised: performance as an SR ensemble with a standard backbone but worse selective classiﬁcation performance than an SR model with a standard backbone.

Original: The big backbone is designed to have twice as many ﬁlters in every convolutional layer and neurons in every fully connected hidden layer as those of the standard VGG-16, which is therefore called Big VGG-16 .
Revised: The big backbone is designed to have twice as many ﬁlters in every convolutional layer and neurons in every fully connected hidden layer as those of the standard VGG-16, which is therefore called Big VGG-16 .

Original: It is easy to see that its number of parameters is approximately 2 2 = 4 times as many as that of standard VGG-16.
Revised: It is easy to see that its number of parameters is approximately 2 2 = 4 times as many as that of standard VGG-16.

Original: We train an
Revised: We train an

Original: SR ensemble of 4 VGG-16s and an SR model with a backbone of Big VGG-16 on CIFAR-10 and show the evaluation results in Figure 7 and Table 5.
Revised: SR ensemble of 4 VGG-16s and an SR model with a backbone of Big VGG-16 on CIFAR-10 and show the evaluation results in Figure 7 and Table 5.

Original: Figure 7 shows that when coverage is high, the ensemble and the big individual model have similar selective risks, and especially, the classiﬁcation error rates (i.e., selective risk of 100% coverage) of the ensemble and the big individual model are similar.
Revised: Figure 7 shows that when coverage is high, the ensemble and the big individual model have similar selective risks, and especially, the classiﬁcation error rates (i.e., selective risk of 100% coverage) of the ensemble and the big individual model are similar.

Original: However, when coverage is low, the big individual model has signiﬁcantly higher selective risk than the ensemble.
Revised: However, when coverage is low, the big individual model has signiﬁcantly higher selective risk than the ensemble.

Original: Table 5 shows that the AURC of Big VGG-16 is much higher than the ensemble of 4 VGG-16s and even higher than SR.
Revised: Table 5 shows that the AURC of Big VGG-16 is much higher than the ensemble of 4 VGG-16s and even higher than SR.

Original: In summary, we show that a selective classiﬁer with a good classiﬁcation performance is not guaranteed to have good selective classiﬁcation performance, so the good selective classiﬁcation performance of the ensemble is not a trivial result of its good classiﬁcation performance.
Revised: In summary, we show that a selective classiﬁer with a good classiﬁcation performance is not guaranteed to have good selective classiﬁcation performance, so the good selective classiﬁcation performance of the ensemble is not a trivial result of its good classiﬁcation performance.

Original: F. T HE E FFECTS OF L ABEL N OISE OF SVHN ON S ELECTIVE C LASSIFIER E NSEMBLES
Revised: F. T HE E FFECTS OF L ABEL N OISE OF SVHN ON S ELECTIVE C LASSIFIER E NSEMBLES

Original: In this section, we compare the effect of label noise of SVHN on the SR ensemble with that on
Revised: In this section, we compare the effect of label noise of SVHN on the SR ensemble with that on

Original: SAT ensemble, whose result might explain the abnormal experimental results (compared to results on other datasets) on SVHN in Section 6.2.
Revised: SAT ensemble, whose result might explain the abnormal experimental results (compared to results on other datasets) on SVHN in Section 6.2.

Original: SVHN is not a clean dataset, and much more label
Revised: SVHN is not a clean dataset, and much more label

Original: [Table] [Table]
Revised: [Table] [Table]

Original: noise can be detected in SVHN than in CIFAR-10 and CIFAR-100.
Revised: noise can be detected in SVHN than in CIFAR-10 and CIFAR-100.

Original: Using the soft label of SAT (Huang et al., 2020), we detect label noise in SVHN, CIFAR-
Revised: Using the soft label of SAT (Huang et al., 2020), we detect label noise in SVHN, CIFAR-

Original: [Table]
Revised: [Table]

Original: and CIFAR-
Revised: and CIFAR-

Original: [Table]
Revised: [Table]

Original: and ﬁnd that
Revised: and ﬁnd that

Original: SVHN has signiﬁcantly more label noise than CIFAR-10 and CIFAR-100.
Revised: SVHN has signiﬁcantly more label noise than CIFAR-10 and CIFAR-100.

Original: The result is presented in the following.
Revised: The result is presented in the following.

Original: In addition, it is known that SAT is robust to label noise (Huang et al., 2020)
Revised: In addition, it is known that SAT is robust to label noise (Huang et al., 2020)

Original: [Table]
Revised: [Table]

Original: while
Revised: while

Original: SR is not so, so we conjecture that the label noise of SVHN is why the SR ensemble is inferior to
Revised: SR is not so, so we conjecture that the label noise of SVHN is why the SR ensemble is inferior to

Original: SAT on SVHN.
Revised: SAT on SVHN.

Original: We detect label noise with the help of the soft label of SAT.
Revised: We detect label noise with the help of the soft label of SAT.

Original: For a sample x i , the soft label of SAT (Huang et al., 2020), t i,y i , is used to measure x i ’s learning difﬁculty.
Revised: For a sample x i , the soft label of SAT (Huang et al., 2020), t i,y i , is used to measure x i ’s learning difﬁculty.

Original: The soft label of SAT is initialized as 1 and updated at every training epoch as below
Revised: The soft label of SAT is initialized as 1 and updated at every training epoch as below

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: e rr ( % ) g
Revised: e rr ( % ) g

Original: i v e n
Revised: i v e n

Original: c
Revised: c

Original: o
Revised: o

Original: v e r a g e o f %
Revised: v e r a g e o f %

Original: CIFARSVHN
Revised: CIFARSVHN

Original: CIFARMRPC
Revised: CIFARMRPC

Original: QNLI
Revised: QNLI

Original: MNLI
Revised: MNLI

Original: [Figure]
Revised: [Figure]

Original: where p θ ( Y | x ) is the predictive probability distribution of the classiﬁer, y i is the label of x i , α is a hyperparameter.
Revised: where p θ ( Y | x ) is the predictive probability distribution of the classiﬁer, y i is the label of x i , α is a hyperparameter.

Original: The smaller the t i,y i is, the lower the true class predictive probability of the classiﬁer on x i during training time, indicating that x i is more difﬁcult to learn.
Revised: The smaller the t i,y i is, the lower the true class predictive probability of the classiﬁer on x i during training time, indicating that x i is more difﬁcult to learn.

Original: By selecting a percentage of samples with the lowest t i,y i , we get the most difﬁcult samples to learn for the classiﬁer, from which we can easily detect label noise manually.
Revised: By selecting a percentage of samples with the lowest t i,y i , we get the most difﬁcult samples to learn for the classiﬁer, from which we can easily detect label noise manually.

Original: In training sets of SVHN, CIFAR-10, and CIFAR-100, we detect label noise manually among the top-0.1% difﬁcult (measured by the soft label of SAT) samples.
Revised: In training sets of SVHN, CIFAR-10, and CIFAR-100, we detect label noise manually among the top-0.1% difﬁcult (measured by the soft label of SAT) samples.

Original: The numbers of mislabeled samples detected in SVHN, CIFAR-10, and CIFAR100 are shown in Table 6.
Revised: The numbers of mislabeled samples detected in SVHN, CIFAR-10, and CIFAR100 are shown in Table 6.

Original: The result shows that
Revised: The result shows that

Original: SVHN has signiﬁcantly more mislabeled samples detected than CIFAR-10 and CIFAR-100, indicating much more label noise in SVHN than in CIFAR-10 and CIFAR-100.
Revised: SVHN has signiﬁcantly more mislabeled samples detected than CIFAR-10 and CIFAR-100, indicating much more label noise in SVHN than in CIFAR-10 and CIFAR-100.

Original: To verify the effect of label noise, the following experiments are designed.
Revised: To verify the effect of label noise, the following experiments are designed.

Original: Firstly, we detect label noise manually among the
Revised: Firstly, we detect label noise manually among the

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: coverage 0.
Revised: coverage 0.

Original: s e
Revised: s e

Original: l e c
Revised: l e c

Original: t
Revised: t

Original: i v e r
Revised: i v e r

Original: i s k 4 VGG-16 Ensemble
Revised: i s k 4 VGG-16 Ensemble

Original: Big VGG-
Revised: Big VGG-

Original: Dataset
Revised: Dataset

Original: [Table]
Revised: [Table]

Original: using the soft label of SAT.
Revised: using the soft label of SAT.

Original: Secondly, we remove the detected mislabeled samples from the original dataset.
Revised: Secondly, we remove the detected mislabeled samples from the original dataset.

Original: The remaining SVHN dataset is called the clean SVHN .
Revised: The remaining SVHN dataset is called the clean SVHN .

Original: Accordingly, the original dataset is called the original SVHN .
Revised: Accordingly, the original dataset is called the original SVHN .

Original: Finally, we retrain and test the SR ensemble and SAT ensemble and compare their test results.
Revised: Finally, we retrain and test the SR ensemble and SAT ensemble and compare their test results.

Original: In the second step, the reason for removing mislabeled samples rather than modifying them is that some samples cannot be classiﬁed even by humans, and some samples are not in the range of categories of SVHN.
Revised: In the second step, the reason for removing mislabeled samples rather than modifying them is that some samples cannot be classiﬁed even by humans, and some samples are not in the range of categories of SVHN.

Original: Thus, the label noise cannot be eliminated by modifying the labels but by removing mislabeled samples.
Revised: Thus, the label noise cannot be eliminated by modifying the labels but by removing mislabeled samples.

Original: The test results of the SR ensemble and SAT ensemble on clean SVHN are shown in Table 7.
Revised: The test results of the SR ensemble and SAT ensemble on clean SVHN are shown in Table 7.

Original: It is not surprising that the AURCs of the SR ensemble and SAT ensemble are signiﬁcantly lower on the clean SVHN than the original SVHN.
Revised: It is not surprising that the AURCs of the SR ensemble and SAT ensemble are signiﬁcantly lower on the clean SVHN than the original SVHN.

Original: Furthermore, on the clean SVHN, when the number of members is 5, the AURC of the SR ensemble is lower than that of SAT ensemble.
Revised: Furthermore, on the clean SVHN, when the number of members is 5, the AURC of the SR ensemble is lower than that of SAT ensemble.

Original: Combined with results on the original SVHN, where the AURC of the SR ensemble is higher than that of SAT ensemble, we conclude that label noise in SVHN is why the SR ensemble has a higher AURC than
Revised: Combined with results on the original SVHN, where the AURC of the SR ensemble is higher than that of SAT ensemble, we conclude that label noise in SVHN is why the SR ensemble has a higher AURC than

Original: SAT ensemble.
Revised: SAT ensemble.

Original: In other words, label noise is why the SR ensemble performs worse in selective classiﬁcation than SAT ensemble on SVHN.
Revised: In other words, label noise is why the SR ensemble performs worse in selective classiﬁcation than SAT ensemble on SVHN.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: In summary, by experiments, we show that the SR ensemble is not as robust to label noise as SAT ensemble, and label noise in SVHN is why the SR ensemble is not as good as SAT ensemble on
Revised: In summary, by experiments, we show that the SR ensemble is not as robust to label noise as SAT ensemble, and label noise in SVHN is why the SR ensemble is not as good as SAT ensemble on

Original: SVHN.
Revised: SVHN.

Original: We construct the clean SVHN , which is SVHN without some mislabeled samples.
Revised: We construct the clean SVHN , which is SVHN without some mislabeled samples.

Original: On the clean SVHN, we compare the SR ensemble with SAT ensemble and ﬁnd that the SR ensemble is superior to SAT ensemble in selective classiﬁcation performance.
Revised: On the clean SVHN, we compare the SR ensemble with SAT ensemble and ﬁnd that the SR ensemble is superior to SAT ensemble in selective classiﬁcation performance.

Original: Combined with former experimental results, we conclude that label noise in SVHN is why the SR ensemble is inferior to SAT on
Revised: Combined with former experimental results, we conclude that label noise in SVHN is why the SR ensemble is inferior to SAT on

Original: SVHN.
Revised: SVHN.

Original: Considering the experimental results on the clean SVHN and previous experimental results on
Revised: Considering the experimental results on the clean SVHN and previous experimental results on

Original: CIFAR-10 and CIFAR-100 (see Table 7 and Table 1), the SR ensemble is superior to SAT ensemble in selective classiﬁcation on clean image classiﬁcation datasets, Thus, SR ensemble is the state-ofthe-art selective classiﬁcation method on clean image classiﬁcation datasets, but is not as robust to label noise as SAT ensemble.
Revised: CIFAR-10 and CIFAR-100 (see Table 7 and Table 1), the SR ensemble is superior to SAT ensemble in selective classiﬁcation on clean image classiﬁcation datasets, Thus, SR ensemble is the state-ofthe-art selective classiﬁcation method on clean image classiﬁcation datasets, but is not as robust to label noise as SAT ensemble.

Original: G A N E XTENSION OF T HEOREM
Revised: G A N E XTENSION OF T HEOREM

Original: This section discusses the lower bound of φ 0 mentioned in Theorem 2.
Revised: This section discusses the lower bound of φ 0 mentioned in Theorem 2.

Original: We aim to calculate φ 0 ’s lower bound without training an ensemble (otherwise, we can measure it directly on the ensemble).
Revised: We aim to calculate φ 0 ’s lower bound without training an ensemble (otherwise, we can measure it directly on the ensemble).

Original: Preliminaries.
Revised: Preliminaries.

Original: The φ 0 can be obtained by solving the following optimization problem.
Revised: The φ 0 can be obtained by solving the following optimization problem.

Original: [Equation]
Revised: [Equation]

Original: where R ens ( φ ) and R ( φ ) are the selective risks of the ensemble and the individual model under coverage φ , respectively.
Revised: where R ens ( φ ) and R ( φ ) are the selective risks of the ensemble and the individual model under coverage φ , respectively.

Original: Supposethat we have allknowledge about the individual model , e.g., the mapping from φ to R is known.
Revised: Suppose we know all about the individual model , e.g., the mapping from φ to R is known.

Original: Since maximizing coverage is equivalent to minimizing the conﬁdence threshold for a ﬁxed model, the original optimization problem can be substituted by
Revised: Since maximizing coverage is equivalent to minimizing the conﬁdence threshold for a ﬁxed model, the original optimization problem can be substituted by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where φ ens ( τ ens ) is coverage of the ensemble with conﬁdence threshold τ ens , and φ ( τ ) is coverage of the individual model given the conﬁdence threshold τ , and then φ 0 = φ ( τ ∗ ) , where τ ∗ is the optimal solution to (42).
Revised: where φ ens ( τ ens ) is coverage of the ensemble with conﬁdence threshold τ ens , and φ ( τ ) is coverage of the individual model given the conﬁdence threshold τ , and then φ 0 = φ ( τ ∗ ) , where τ ∗ is the optimal solution to (42).

Original: Assume that R is a monotone increasing function and that φ is a monotone decreasing function 5 , then it is easy to show, using proof by contradiction, that (42) can be further transformed into
Revised: Assume that R is a monotone increasing function and that φ is a monotone decreasing function 5 , then it is easy to show, using proof by contradiction, that (42) can be further transformed into

Original: [Equation]
Revised: [Equation]

Original: To solve (43), we need more information about the ensemble.
Revised: To solve (43), we need more information about the ensemble.

Original: Besides the number of classes K , assume that we know: an oracle that tells whether a sample is deﬁnite; M , the number of member models; and B , the upper bound of p Π k 1 ,..., Π kM ( ·| A ) for all k ∈ { 1 , 2 , ..., K } .
Revised: Besides the number of classes K , assume that we know: an oracle that tells whether a sample is deﬁnite; M , the number of member models; and B , the upper bound of p Π k 1 ,..., Π kM ( ·| A ) for all k ∈ { 1 , 2 , ..., K } .

Original: It is natural to know K and M , and we need the oracle and B because they provide critical information about the ensemble’s behavior.
Revised: It is natural to know K and M , and we need the oracle and B because they provide critical information about the ensemble’s behavior.

Original: The oracle can be implemented by an ensemble with M (cid:48)
Revised: The oracle can be implemented by an ensemble with M (cid:48)

Original: ( M (cid:48) (cid:28) M ) members.
Revised: ( M (cid:48) (cid:28) M ) members.

Original: Eliminating the Unknowns.
Revised: Eliminating the Unknowns.

Original: We are now committed to translating the unknowns in the (43) into known quantities.
Revised: We are now committed to translating the unknowns in the (43) into known quantities.

Original: Firstly, we eliminate the unknowns in the ﬁrst constraint.
Revised: Firstly, we eliminate the unknowns in the ﬁrst constraint.

Original: According to (20) and
Revised: According to (20) and

Original: Lemma 4, it is easy to prove (by an integral)
Revised: Lemma 4, it is easy to prove (by an integral)

Original: [Equation]
Revised: [Equation]

Original: 44) where C and C ens are the conﬁdence scores of the individual model and the ensemble, respectively,
Revised: (44) where C and C ens are the conﬁdence scores of the individual model and the ensemble, respectively,

Original: A / D represents the event that the input sample is ambiguous/deﬁnite, β = K · M M − 1 · B .
Revised: A / D represents the event that the input sample is ambiguous/deﬁnite, β = K · M M − 1 · B .

Original: Combining
Revised: Combining

Original: (44) with φ ens =
Revised: (44) with φ ens =

Original: Pr( C ≥ τ ens , A
Revised: Pr( C ≥ τ ens , A

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: Thus, we can intensify the ﬁrst constraint as
Revised: Thus, we can intensify the ﬁrst constraint as

Original: [Equation]
Revised: [Equation]

Original: Secondly, we eliminate the unknowns in the second constraint.
Revised: Secondly, we eliminate the unknowns in the second constraint.

Original: according to (41), (35) is a sufﬁcient condition of R ens < R .
Revised: according to (41), (35) is a sufﬁcient condition of R ens < R .

Original: We rewrite (35) as
Revised: We rewrite (35) as

Original: [Equation]
Revised: [Equation]

Original: where C ens is the conﬁdence score of the ensemble,  Err represents the event that the individual model makes a error prediction.
Revised: where C ens is the conﬁdence score of the ensemble, and Err represents the event that the individual model makes an error prediction.

Original: Note that the ﬁrst term in the second constraint of (47) contains
Revised: Note that the ﬁrst term in the second constraint of (47) contains

Original: C ens , which is unknown, so we cannot directly replace (43)’s second constraint with (47).
Revised: C ens , which is unknown, so we cannot directly replace (43)’s second constraint with (47).

Original: We eliminate C ens as follows:
Revised: We eliminate C ens as follows:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is due to (44), and the second inequality is due to (45) and (46).
Revised: where the ﬁrst inequality is due to (44), and the second inequality is due to (45) and (46).

Original: Thus, a sufﬁcient condition of (47) is
Revised: Thus, a sufﬁcient condition of (47) is

Original: [Equation]
Revised: [Equation]

Original: with which we replace the second constraint of (43).
Revised: with which we replace the second constraint of (43).

Original: In summary, we can intensify the constraints of (43) and obtain the following optimization problem that does not contain the unknowns.
Revised: In summary, we can intensify the constraints of (43) and obtain the following optimization problem that does not contain the unknowns.

Original: It is easy to see that  optimal solution to (49) is a upper bound of that to (43).
Revised: It is easy to see that the optimal solution to (49) is an upper bound of that to (43).

Original: [Equation]
Revised: [Equation]

Original: Further Simpliﬁcation and Final Result.
Revised: Further Simpliﬁcation and Final Result.

Original: It is easy to show, by proof of contraction, that the ﬁrst constraint of (49) can be substituted by
Revised: It is easy to show, by proof of contraction, that the ﬁrst constraint of (49) can be substituted by

Original: [Equation]
Revised: [Equation]

Original: Thus, the second constraint of (49) can be simpliﬁed as
Revised: Thus, the second constraint of (49) can be simpliﬁed as

Original: [Equation]
Revised: [Equation]

Original: ⇔Pr
Revised: ⇔

Original: [Equation]
Revised: [Equation]

Original: A)
Revised: A

Original: [Equation]
Revised: [Equation]

Original: β
Revised: ⇔

Original: [Equation]
Revised: [Equation]

Original: 1 − τ [Equation] | D [Equation]
Revised: A

Original: C ≥ τ

Original: [Equation]
Revised: [Equation]

Original: Pr

Original: [Equation] ≥ [Equation] < R
Revised: β (

Original: [Equation]
Revised: [Equation]

Original: φ [Equation] ⇔ Pr [Equation]
Revised: τ

Original: A )

Original: [Equation]
Revised: [Equation]

Original: β
Revised: ) M

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: τ
Revised: Pr( Err

Original: [Equation]
Revised: [Equation]

Original: [Equation] Pr( Err
Revised: D

Original: [Equation]
Revised: [Equation]

Original: C ≥ τ
Revised: C ≥ τ

Original: [Equation]
Revised: [Equation]

Original: ) <
Revised: ) <

Original: R ( φ ( τ )) φ ( τ ) ⇔ Pr( A ) · β (1 − τ ens )
Revised: R ( φ ( τ )) φ ( τ ) ⇔ Pr( A ) · β (1 − τ ens )

Original: M + Pr( Err, D, C ≥ τ ens ) <
Revised: M + Pr( Err, D, C ≥ τ ens ) <

Original: Pr( Err, C ≥ τ ) .
Revised: Pr( Err, C ≥ τ ) [Equation]

Original: Thus, the ﬁnal version of the optimization problem with respect to τ is
Revised: Thus, the ﬁnal version of the optimization problem with respect to τ is

Original: [Equation]
Revised: [Equation]

Original: ( Err, C ≥ τ ) .
Revised: < Pr( Err, C ≥ τ ) .

Original:  if τ ens is not None and V ERIFY S ECOND C ONSTRAINT ( τ , τ ens , θ , D , Ω , K , M , B ) is True then rightelse left = 
Revised: τ S if τ ens is not None and V ERIFY S ECOND C ONSTRAINT ( τ , τ ens , θ , D , Ω , K , M , B ) is True then right = else

Original:  τ τ ∗ = 
Revised: left = τ τ ∗ = (

Original: + right return 1 N 
Revised: + right /return 1 N (cid:80) Ni

Original: Ni =1{+=
+} I {  ) ≥ τ ∗ } //  is the conﬁdence of  on sample x i .
Revised: {+=
+} I { C ( i ; θ ) ≥ τ ∗ } // C ( x i ; θ ) is the conﬁdence of θ on sample x i .

Original: Suppose φ 0 = φ ( τ 0 ) , since τ 0 is the optimal solution to (43), the optimal solution to (50) (denoted as τ ∗ ) provides an upper bound of τ 0 .
Revised: Suppose φ 0 = φ ( τ 0 ) , since τ 0 is the optimal solution to (43), the optimal solution to (50) (denoted as τ ∗ ) provides an upper bound of τ 0 .

Original: Thus, considering φ is a monotone decreasing function with respect to τ , φ ( τ ∗ ) is a lower bound of φ 

Revised: Thus, considering φ is a monotone decreasing function of τ , φ ( τ ∗ ) is a lower bound of φ

Original: = φ ( τ 0 ) .
Revised: = φ ( τ 0 ) .

Original: Algorithm.
Revised: Algorithm.

Original: We design Algorithm 1 to search for the solution to (50) and then obtain the lower bound of φ 0 .
Revised: We design Algorithm 1 to search for the solution to (50) and then obtain the lower bound of φ 0 .

Original: Since τ ens is determined by τ (the ﬁrst constraint of (50)), (50) can be reduced to a one-dimensional search problem.
Revised: Since τ ens is determined by τ (the ﬁrst constraint of (50)), (50) can be reduced to a one-dimensional search problem.

Original: We adopt a binary searchin our algorithm for efﬁciency, although this method might provide a suboptimal solution.
Revised: Our algorithm adopts a binary search for efﬁciency, although this method might provide a suboptimal solution.

Original: The procedure of Algorithm 1 in each iteration of the binary search is as follows.
Revised: The procedure of Algorithm 1 in each iteration of the binary search is as follows.

Original: Given current τ , Algorithm 1 determines τ ens using S EARCH F OR T AU E NS (see Algorithm 2), a procedure that searches for τ ens ∈ [0 , 1] using binary search s.t.
Revised: Given current τ , Algorithm 1 determines τ ens using S EARCH F OR T AU E NS (see Algorithm 2), a procedure that searches for τ ens ∈ [0 , 1] using binary search s.t.

Original: φ ( τ ) =
Revised: φ ( τ ) =

Original: Pr( C ≥ τ ens , D ) .
Revised: Pr( C ≥ τ ens , D ) .

Original: Note that τ
Revised: Note that τ

Original: ens might not exist, as long as τ is so low
Revised: ens might not exist, as long as τ is so low

Original: that φ ( τ ) >
Revised: that φ ( τ ) >

Original: Pr( D ) = sup
Revised: Pr( D ) = sup

Original: τ ens Pr( C ≥ τ ens , D ) .
Revised: τ ens Pr( C ≥ τ ens , D ) .

Original: This problem will be addressed shortly.
Revised: This problem will be addressed shortly.

Original: Algorithm 1 exams whether τ ens exists.
Revised: Algorithm 1 exams whether τ ens exists.

Original: If τ ens exists, Algorithm 1 then exams whether the second constraints of (50) holds for current τ and τ ens , which is implemented by V ERIFYS ECOND C ONSTRAINT (see Algorithm 3).
Revised: If τ ens exists, Algorithm 1 then examines whether the second constraint of (50) holds for current τ and τ ens , which is implemented by V ERIFY S ECOND C ONSTRAINT (see Algorithm 3).

Original: If τ ens exists and the second constraint holds, Algorithm 1 searches for a smaller τ in the left half feasible area; otherwise, Algorithm 1 searches for a greater τ in the right half feasible area.
Revised: If τ ens exists and the second constraint holds, Algorithm 1 searches for a smaller τ in the left half feasible area; otherwise, Algorithm 1 searches for a greater τ in the right half feasible area.

Original: Once the binary search completes and outputs τ ∗ , Algorithm 1 returns the coverage of θ with conﬁdence threshold τ ∗ .
Revised: Once the binary search completes and outputs τ ∗ , Algorithm 1 returns the coverage of θ with conﬁdence threshold τ ∗ .

Original: An Example.
Revised: An Example.

Original: To show that Algorithm 1 works in reality, we run this algorithm on CIFAR-10, using the same individual model as Section 6.
Revised: To show that Algorithm 1 works in reality, we run this algorithm on CIFAR-10, using the same individual model as Section 6.

Original: In this example, K = 10 , M = 5 , the oracle is implemented by another ensemble with two individual models (the oracle outputs True if and only if the STD over member models’ predictive distributions < 10 − 3 ).
Revised: In this example, K = 10 , M = 5 , the oracle is implemented by another ensemble with two individual models (the oracle outputs True if and only if the STD over member models’ predictive distributions < 10 − 3 ).

Original: Note that it is difﬁcult to estimateof B .
Revised: Note that it is difﬁcult to estimate B .

Original: On the one hand, we need to train an ensemble with M models to estimate B , which is costly.
Revised: On the one hand, we need to train an ensemble with M models to estimate B , which is costly.

Original: On the other hand, the domain of p Π k 1 ,..., Π kM ( ·| A ) has high dimension, so the observed data points are sparse in this domain, which makes the estimation of B more difﬁcult.
Revised: On the other hand, the domain of p Π k 1 ,..., Π kM ( ·| A ) has high dimension, so the observed data points are sparse in this domain, which makes the estimation of B more difﬁcult.

Original: Thus, we do not estimate B but try several hypothetical values of B to see at what B the lower bound of φ 0 is big.
Revised: Thus, we do not estimate B but try several hypothetical values of B to see at what B the lower bound of φ 0 is big.

Original: With different B s, we obtain different lower bounds of φ 

Revised: With different B s, we obtain different lower bounds of φ

Original: as Table 8 shows.
Revised: as Table 8 shows.

Original: We can see that when B ≤ 10 8 , the lower bound of φ 0 is greater than 50%, which indicates that Algorithm 1 may be robust to the choice of B . is the on x if φ > 1 N (cid:80)
Revised: We can see that when B ≤ 10 8 , the lower bound of φ 0 is greater than 50%, which indicates that Algorithm 1 may be robust to the choice of B . is the on x if φ > 1 N (cid:80)

Original: Ni =1 Ω( x i ) then return None left =
Revised: Ni =1 Ω( x i ) then return None left =

Original: right
Revised: right

Original: (cid:15) = 10 −while right − left do τ ens =
Revised: (cid:15) = 10 −while right − left do τ ens =

Original: ( left + right
Revised: ( left + right

Original: /if 1 N (cid:80) Ni
Revised: /if 1 N (cid:80) Ni

Original: =

Revised: =

Original: I { C ( x i ; θ ) ≥ τ ens } · Ω( x i ) <
Revised: I { C ( x i ; θ ) ≥ τ ens } · Ω( x i ) <

Original: φ then right
Revised: φ then right

Original: =
Revised: =

Original: τ
Revised: τ

Original: ens else left = τ ens return ( left + right ) /
Revised: ens else left = τ ens return ( left + right ) /

Original: Input: the conﬁdence threshold τ ; τ ens ; the individual model θ ; the test set D i , y i ) } Ni =1 ; the oracle Ω : X → { 0 , 1 } that tells whether a sample is deﬁnite; the number of classes
Revised: Input: the conﬁdence threshold τ ; τ ens ; the individual model θ ; the test set D i , y i ) } Ni =1 ; the oracle Ω : X → { 0 , 1 } that tells whether a sample is deﬁnite; the number of classes

Original: K ; the number of member models M ; and B , the upper bound of p Π k 1 ,..., Π kM ( ·| A ) for all k ∈ { 1 , 2 , ..., K } .
Revised: K ; the number of member models M ; and B , the upper bound of p Π k 1 ,..., Π kM ( ·| A ) for all k ∈ { 1 , 2 , ..., K } .

Original: Output
Revised: Output

Original: P A = 1 N (cid:80) Ni =1 [1 − Ω( x i )]
Revised: P A = 1 N (cid:80) Ni =1 [1 − Ω( x i )]

Original: β = K · M M − 1 · B leftHandSide = P A · β (1 − τ ens )
Revised: β = K · M M − 1 · B leftHandSide = P A · β (1 − τ ens )

Original: M + 1 N (cid:80)
Revised: M + 1 N (cid:80)

Original: Ni
Revised: Ni

Original: =

Revised: =

Original: I { f ( x i ; θ ) (cid:54) =
Revised: I { f ( x i ; θ ) (cid:54) =

Original: y i } · Ω( x i
Revised: y i } · Ω( x i

Revised: [Equation]

Revised: rightHandSide

Original: [Equation]
Revised: [Equation]

Original: This example also indicates the relationship between  ensemble’s diversity and its selective classiﬁcation performance.
Revised: This example also indicates the relationship between the ensemble’s diversity and its selective classiﬁcation performance.

Original: Since an ensemble with a smaller B seems to have more diversity over ambiguous samples, the result in Table 8 suggests that as long as the ensemble has enough diversity over ambiguous samples, the ensemble is guaranteed to have a lower selective risk than the individual model under a considerable range of coverage
Revised: Since an ensemble with a smaller B seems to have more diversity over ambiguous samples, the result in Table 8 suggests that as long as the ensemble has enough diversity over ambiguous samples, the ensemble is guaranteed to have a lower selective risk than the individual model under a considerable range of coverage.

Original: [Table]
Revised: [Table]

Original: [Table]