Original: T OWARDS B ETTER S ELECTIVE C LASSIFICATION
Revised: T OWARDS B ETTER S ELECTIVE C LASSIFICATION

Original: Anonymous authors
Revised: ∗ Mila & Borealis AI Mohamed Osama Ahmed Borealis AI

Original: Paper under double-blind review
Revised: Hossein Hajimirsadeghi Borealis AI Amir Abdi Microsoft

Original: A BSTRACT
Revised: A BSTRACT

Original: We tackle the problem of Selective Classification where the objective is to achieve the best performance on a predetermined ratio (coverage) of the dataset.
Revised: We tackle the problem of Selective Classification where the objective is to achieve the best performance on a predetermined ratio (coverage) of the dataset.

Original: Recent state-of-the-art selective methods come with architectural changes either via introducing a separate selection head or an extra abstention logit.
Revised: Recent state-of-the-art selective methods come with architectural changes either via introducing a separate selection head or an extra abstention logit.

Original: In this paper, we challenge the aforementioned methods and confirm that the superior performance of state-of-the-art methods is owed to training a more generalizable classifier rather than their proposed selection mechanisms.
Revised: In this paper, we challenge the aforementioned methods. The results suggest that the superior performance of state-of-the-art methods is owed to training a more generalizable classifier rather than their proposed selection mechanisms.

Original: We argue that the best performing selection mechanism should instead be rooted in the classifier itself.
Revised: We argue that the best performing selection mechanism should instead be rooted in the classifier itself.

Original: Our proposed selection strategy uses the classification scores and achieves better results by a significant margin, consistently, across all coverages and all datasets, without any added compute cost.
Revised: Our proposed selection strategy uses the classification scores and achieves better results by a significant margin, consistently, across all coverages and all datasets, without any added compute cost.

Original: Furthermore, inspired by semi-supervised learning, we propose an entropy-based regularizer that improves the performance of selective classification methods.
Revised: Furthermore, inspired by semi-supervised learning, we propose an entropy-based regularizer that improves the performance of selective classification methods.

Original: Our proposed selection mechanism with the proposed entropy-based regularizer achieves new state-of-the-art results.
Revised: Our proposed selection mechanism with the proposed entropy-based regularizer achieves new state-of-the-art results.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A model’s ability to abstain from a decision when lacking confidence is essential in mission-critical applications.
Revised: A model’s ability to abstain from a decision when lacking confidence is essential in mission-critical applications.

Original: This is known as the Selective Prediction problem setting.
Revised: This is known as the Selective Prediction problem setting.

Original: The abstained and uncertain samples can be flagged and passed to a human expert for manual assessment, which, in turn, can improve the re-training process.
Revised: The abstained and uncertain samples can be flagged and passed to a human expert for manual assessment, which, in turn, can improve the re-training process.

Original: This is crucial in problem settings where confidence is critical or an incorrect prediction can have significant consequences such as in the financial, medical, or autonomous driving domains.
Revised: This is crucial in problem settings where confidence is critical or an incorrect prediction can have significant consequences such as in the financial, medical, or autonomous driving domains.

Original: Several papers have tried to address this problem by estimating the uncertainty in the prediction.
Revised: Several papers have tried to address this problem by estimating the uncertainty in the prediction.

Original: Gal & Ghahramani (2016) proposed using MC-dropout, Lakshminarayanan et al.
Revised: Gal & Ghahramani (2016) proposed using MC-dropout. Lakshminarayanan et al.

Original: (2017) proposed to use an ensemble of models, while methods such as Dusenberry et al.(2020) and
Revised: (2017) proposed to use an ensemble of models. Dusenberry et al.

Original:  Maddox et al.
Revised: and Maddox et al.

Original: (2019) are examples of work using Bayesian deep learning to estimate a probability distribution of the output.
Revised: (2019) are examples of work using Bayesian deep learning.

Original: These methods are either expensive to train or require lots of tuning for acceptable results.
Revised: These methods, however, are either expensive to train or require lots of tuning for acceptable results.

Original: In this paper, we focus on the Selective Classification problem setting where a classifier has the option to abstain from making predictions.
Revised: In this paper, we focus on the Selective Classification problem setting where a classifier has the option to abstain from making predictions.

Original: Models that come with an abstention option and tackle the selective prediction problem setting are naturally called selective models .
Revised: Models that come with an abstention option and tackle the selective prediction problem setting are naturally called selective models .

Original: Different selection approaches have been suggested such as incorporating a selection head Geifman & El-Yaniv (2019) or an abstention logit (Huang et al., 2020; Ziyin et al., 2019).
Revised: Different selection approaches have been suggested such as incorporating a selection head Geifman & El-Yaniv (2019) or an abstention logit (Huang et al., 2020; Ziyin et al., 2019).

Original: In either case, a threshold is set such that selection and abstention values above or below the threshold decide the selection action.
Revised: In either case, a threshold is set such that selection and abstention values above or below the threshold decide the selection action.

Original: SelectiveNet
Revised: SelectiveNet

Original: Geifman & El-Yaniv (2019) proposes to learn a model comprising of a selection head and a prediction head where the values returned by the selection head determines whether the datapoint is selected for prediction or not.
Revised: Geifman & El-Yaniv (2019) proposes to learn a model comprising of a selection head and a prediction head where the values returned by the selection head determines whether the datapoint is selected for prediction or not.

Original: Huang et al.
Revised: Huang et al.

Original: (2020) and Ziyin et al.
Revised: (2020) and Ziyin et al.

Original: (2019) introduced an additional abstention logit for classification settings where the output of the additional logit determines whether the model abstains from making predictions on the sample.
Revised: (2019) introduced an additional abstention logit for classification settings where the output of the additional logit determines whether the model abstains from making predictions on the sample.

Original: The promising results of these works suggest that the selection mechanism should focus on the output of an external head/logit.
Revised: The promising results of these works suggest that the selection mechanism should focus on the output of an external head/logit.

Original: On the contrary, in this work, we argue that the selection mechanism should be rooted in the classifier itself.
Revised: On the contrary, in this work, we argue that the selection mechanism should be rooted in the classifier itself.

Original: The results of our rigorously conducted experiments show that (1) the superior performance of the state-of-the-art methods is owed to training a more generalizable classifier rather than their proposed external head/logit selection mechanisms.
Revised: The results of our rigorously conducted experiments show that (1) the superior performance of the state-of-the-art methods is owed to training a more generalizable classifier rather than their proposed external head/logit selection mechanisms.

Original: These results suggest that future work in selective classification (i) should aim to learn a more generalizable classifier and (ii) the selection mechanism should be based on the classifier itself rather than the recent research directions of architecture modifications for an external logit/head.
Revised: These results suggest that future work in selective classification (i) should aim to learn a more generalizable classifier and (ii) the selection mechanism should be based on the classifier itself rather than the recent research directions of architecture modifications for an external logit/head.

Original: (2) We highlight a connection between selective classification and semi-supervised learning.
Revised: (2) We highlight a connection between selective classification and semi-supervised learning.

Original: To the best of our knowledge, this has has not been explored before.
Revised: To the best of our knowledge, this has has not been explored before.

Original: We show that entropy-minimization regularization, a common technique in semi-supervised learning, significantly improves the performance of the state-of-the-art selective classification method.
Revised: We show that entropy-minimization regularization, a common technique in semi-supervised learning, significantly improves the performance of the state-of-the-art selective classification method.

Original: The promising results suggest that additional research is warranted to explore the relationship between these two research directions.
Revised: The promising results suggest that additional research is warranted to explore the relationship between these two research directions.

Original: From a practical perspective, (3) we propose a selection mechanism that outperforms the original selection mechanism of state-of-the-art methods.
Revised: From a practical perspective, (3) we propose a selection mechanism that outperforms the original selection mechanism of state-of-the-art methods.

Original: Furthermore, this method can be immediately applied to an already deployed selective classification model and instantly improve performance at no additional cost.
Revised: Furthermore, this method can be immediately applied to an already deployed selective classification model and instantly improve performance at no additional cost.

Original: (4) We show a selective classifier trained with the entropy-regularised loss and with selection according to the classification scores achieves new state-of-the-art results by a significant margin (up to 80% relative improvement).
Revised: (4) We show a selective classifier trained with the entropy-regularised loss and with selection according to the classification scores achieves new state-of-the-art results by a significant margin (up to 80% relative improvement).

Original: (5) Going beyond the already-saturated datasets often used for Selective Classification research, wealso include 2 larger datasets:  Imagenet100 to test the methods on a wide range of coverages and ImagenetSubset to test the scalability of the methods.
Revised: (5) Going beyond the already-saturated datasets often used for Selective Classification research, we include results on larger datasets: StanfordCars, Food101, Imagenet, and Imagenet100 to test the methods on a wide range of coverages and ImagenetSubset to test the scalability of the methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The option to reject a prediction has been explored in depth in various learning algorithms not limited to neural networks.
Revised: The option to reject a prediction has been explored in depth in various learning algorithms not limited to neural networks.

Original: Primarily, Chow (Chow, 1970) introduced a cost-based rejection model and analysed the error-reject trade-off.
Revised: Primarily, Chow (Chow, 1970) introduced a cost-based rejection model and analysed the error-reject trade-off.

Original: There has been significant study in rejection in Support Vector
Revised: There has been significant study in rejection in Support Vector

Original: Machines (Bartlett & Wegkamp, 2008; Fumera & Roli, 2002; Wegkamp, 2007; Wegkamp & Yuan, 2011).
Revised: Machines (Bartlett & Wegkamp, 2008; Fumera & Roli, 2002; Wegkamp, 2007; Wegkamp & Yuan, 2011).

Original: The same is true for nearest neighbours (Hellman, 1970) and boosting (Cortes et al., 2016) literature.
Revised: The same is true for nearest neighbours (Hellman, 1970) and boosting (Cortes et al., 2016).

Original: In 1989, LeCun et al.
Revised: In 1989, LeCun et al.

Original: (1989) proposed a rejection strategy for neural networks based on the most activated output logit, second most activated output logit, and the difference between the activated output logits.
Revised: (1989) proposed a rejection strategy for neural networks based on the most activated output logit, second most activated output logit, and the difference between the activated output logits.

Original: Geifman & El-Yaniv (2017) presented a technique to achieve a target risk with a certain probability for a given confidence-rate function.
Revised: Geifman & El-Yaniv (2017) presented a technique to achieve a target risk with a certain probability for a given confidence-rate function.

Original: As examples of confidence-rate functions, the authors suggested selecting according to Softmax Response and MC-Dropout as selection mechanisms for a vanilla classifier.
Revised: As examples of confidence-rate functions, the authors suggested selecting according to Softmax Response and MC-Dropout as selection mechanisms for a vanilla classifier.

Original: We build on this idea to demonstrate that Softmax Response, if utilized correctly, is the highest performing selection mechanism in the selective classification settings.
Revised: We build on this idea to demonstrate that Softmax Response, if utilized correctly, is the highest performing selection mechanism in the selective classification settings.

Original: Beyond selective classification, max-logit (Softmax Response) has also been used in anomaly detection (Dietterich &
Revised: Beyond selective classification, max-logit (Softmax Response) has also been used in anomaly detection (Hendrycks &

Original:  Guyer, 2022).
Revised: Gimpel, 2016; Dietterich & Guyer, 2022).

Original: Future work focused on architectural changes and selecting according to a separately computed head/logit with their own parameters.
Revised: Future work focused on architectural changes and selecting according to a separately computed head/logit with their own parameters.

Original: The same authors, Geifman & El-Yaniv (2019) later proposed
Revised: The same authors, Geifman & El-Yaniv (2019) later proposed

Original: SelectiveNet (see Section 3.2.1), a three-headed model, comprising of heads for selection, prediction, and auxiliary prediction.
Revised: SelectiveNet (see Section 3.2.1), a three-headed model, comprising of heads for selection, prediction, and auxiliary prediction.

Original: Deep Gamblers (Ziyin et al., 2019) (see Appendix A.1) and Self-Adaptive
Revised: Deep Gamblers (Ziyin et al., 2019) (see Appendix A.1) and Self-Adaptive

Original: Training (Huang et al., 2020) (see Section 3.3.1) propose a ( C + 1) -way classifier, where C is the number of classes and the additional logit represents abstention.Ziyin et al.
Revised: Training (Huang et al., 2020) (see Section 3.3.1) propose a ( C + 1) -way classifier, where C is the number of classes and the additional logit represents abstention.

Original: compared selecting according to the model’s uncertainty, in particular, the prediction entropy, for a vanilla classifier as a baseline, but reported inferior results than Deep Gamblers.

Original: In contrast, in this work, we explain how selecting via entropy and max-logit can work as a proxy to select samples which could potentially minimise the cross entropy loss.
Revised: In contrast, in this work, we explain how selecting via entropy and max-logit can work as a proxy to select samples which could potentially minimise the cross entropy loss.

Original: In general, we report the surprising results that the selection head of the SelectiveNet and the abstention logits in Deep Gamblers and Self-Adaptive 
Revised: In general, we report the surprising results that the selection head of the SelectiveNet and the abstention logits in Deep Gamblers and Self-Adaptive Training are suboptimal selection mechanisms.

Original: Training are suboptimal selection mechanisms and their previously reported good performance is rooted in their optimization process converging to a more generalizable model.
Revised: Furthermore, their previously reported good performance is rooted in their optimization process converging to a more generalizable model.

Original: Another line of work which tackles the selective classification is that of cost-sensitive classification (Charoenphakdee et al., 2021).
Revised: Another line of work which tackles the selective classification is that of cost-sensitive classification (Charoenphakdee et al., 2021).

Original: However, the introduction of the target coverage adds a new variable and changes the mathematical formulation.
Revised: However, the introduction of the target coverage adds a new variable and changes the mathematical formulation.

Original: Other works have proposed to perform classification in conjunction with expert decision makers (Mozannar & Sontag, 2020).
Revised: Other works have proposed to perform classification in conjunction with expert decision makers (Mozannar & Sontag, 2020).

Original: In this work, we also highlight a connection between semi-supervised learning and selective classification, which, to the best of our knowledge, has not been explored before.
Revised: In this work, we also highlight a connection between semi-supervised learning and selective classification, which, to the best of our knowledge, has not been explored before.

Original: As a result, we propose an entropy-regularized loss function in the Selective Classification settings to further improve the performance of the Softmax Response selection mechanism.
Revised: As a result, we propose an entropy-regularized loss function in the Selective Classification settings to further improve the performance of the Softmax Response selection mechanism.

Original: However, entropy minimization objectives have been widely used for Unsupervised Learning (Long et al., 2016), Semi-Supervised
Revised: However, entropy minimization objectives have been widely used for Unsupervised Learning (Long et al., 2016), Semi-Supervised

Original: Learning (Grandvalet & Bengio, 2004), and Domain Adaptation (Vu et al., 2019; Wu et al., 2020).
Revised: Learning (Grandvalet & Bengio, 2004), and Domain Adaptation (Vu et al., 2019; Wu et al., 2020).

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section, we introduce the Selective Classification problem.
Revised: In this section, we introduce the Selective Classification problem.

Original: Additionally, we describe the topthree methods for Selective Classification: SelectiveNet (Geifman & El-Yaniv, 2019), Self-Adaptive
Revised: Additionally, we describe the top methods for Selective Classification.

Original: Training (Huang et al., 2020), and Deep Gamblers (Ziyin et al., 2019).To the best of our knowledge, Self-Adaptive Training  achieves the best performance on the selective classification datatests.
Revised: To the best of our knowledge, Self-Adaptive Training (Huang et al., 2020) achieves the best performance on the selective classification datatests.

Original: 3. P ROBLEM S ETTING : S ELECTIVE C LASSIFICATION
Revised: 3. P ROBLEM S ETTING : S ELECTIVE C LASSIFICATION

Original: The selective prediction task can be formulated as follows.
Revised: The selective prediction task can be formulated as follows.

Original: Let X be the feature space, Y be the label space, and P ( X , Y ) represent the data distribution over X × Y .
Revised: Let X be the feature space, Y be the label space, and P ( X , Y ) represent the data distribution over X × Y .

Original: A selective model comprises of a prediction function f : X → Y and a selection function g : X → { 0 , 1 } .
Revised: A selective model comprises of a prediction function f : X → Y and a selection function g : X → { 0 , 1 } .

Original: The selective model decides to make predictions when g ( x ) = 1 and abstains from making predictions when g ( x ) = 0 .
Revised: The selective model decides to make predictions when g ( x ) = 1 and abstains from making predictions when g ( x ) = 0 .

Original: The objective is to maximise the model’s predictive performance for a given target coverage c target ∈ [0 , 1] , where coverage is the proportion of the selected samples.
Revised: The objective is to maximise the model’s predictive performance for a given target coverage c target ∈ [0 , 1] , where coverage is the proportion of the selected samples.

Original: The selected set is defined as { x : g ( x ) = 1 } .
Revised: The selected set is defined as { x : g ( x ) = 1 } .

Original: Formally, an optimal selective model, parameterised by θ ∗ and ψ ∗ , would be the following:
Revised: Formally, an optimal selective model, parameterised by θ ∗ and ψ ∗ , would be the following:

Original: [Equation]
Revised: [Equation]

Original: where E P [ l ( f θ ( x ) , y ) · g ψ ( x )] is the selective risk.
Revised: where E P [ l ( f θ ( x ) , y ) · g ψ ( x )] is the selective risk.

Original: Naturally, higher coverages are correlated with higher selective risks.
Revised: Naturally, higher coverages are correlated with higher selective risks.

Original: In practice, instead of a hard selection function g ψ ( x ) , existing methods aim to learn a soft selection function ¯ g ψ : X → R such that larger values of ¯ g ψ ( x ) indicate the datapoint should be selected for prediction.
Revised: In practice, instead of a hard selection function g ψ ( x ) , existing methods aim to learn a soft selection function ¯ g ψ : X → R such that larger values of ¯ g ψ ( x ) indicate the datapoint should be selected for prediction.

Original: At test time, a threshold τ is selected for a coverage c such that
Revised: At test time, a threshold τ is selected for a coverage c such that

Original: [Equation]
Revised: [Equation]

Original: In this setting, the selected (covered) dataset is defined as { x : ¯ g ψ ( x ) ≥ τ } .
Revised: In this setting, the selected (covered) dataset is defined as { x : ¯ g ψ ( x ) ≥ τ } .

Original: The process of selecting the threshold τ is known as calibration .
Revised: The process of selecting the threshold τ is known as calibration .

Original: 3. A PPROACH : L EARN TO S ELECT 3.2. S ELECTIVE N ET
Revised: 3. A PPROACH : L EARN TO S ELECT 3.2. S ELECTIVE N ET

Original: SelectiveNet (Geifman & El-Yaniv, 2019) is a three-headed network proposed for selective learning.
Revised: SelectiveNet (Geifman & El-Yaniv, 2019) is a three-headed network proposed for selective learning.

Original: A SelectiveNet model has three output heads designed for selection ¯ g , prediction f , and auxiliary prediction h .
Revised: A SelectiveNet model has three output heads designed for selection ¯ g , prediction f , and auxiliary prediction h .

Original: The selection head infers the selective score of each sample, as a value between 0 to 1, and is implemented with a sigmoid activation function.
Revised: The selection head infers the selective score of each sample, as a value between 0 to 1, and is implemented with a sigmoid activation function.

Original: The auxiliary prediction head is trained with a standard (non-selective) loss function.
Revised: The auxiliary prediction head is trained with a standard (non-selective) loss function.

Original: Given a batch { ( x i , y i ) } mi =1 , where y i is the label, the model is trained to minimise the loss L where it is defined as:
Revised: Given a batch { ( x i , y i ) } mi =1 , where y i is the label, the model is trained to minimise the loss L where it is defined as:

Original: [Equation]
Revised: [Equation]

Original: where ℓ is any standard loss function.
Revised: where ℓ is any standard loss function.

Original: In Selective Classification, ℓ is the Cross Entropy loss function.
Revised: In Selective Classification, ℓ is the Cross Entropy loss function.

Original: The coverage loss L c encourages the model to achieve the desired coverage and ensures ¯ g ( x i ) >for at least c target proportion of the batch samples.
Revised: The coverage loss L c encourages the model to achieve the desired coverage and ensures ¯ g ( x i ) >for at least c target proportion of the batch samples.

Original: The selective loss L selective discounts the weight of difficult samples via the soft selection value ¯ g ( x ) term encouraging the model to focus more on easier samples which the model is more confident about.
Revised: The selective loss L selective discounts the weight of difficult samples via the soft selection value ¯ g ( x ) term encouraging the model to focus more on easier samples which the model is more confident about.

Original: The auxiliary loss L aux ensures that all samples, regardless of their selective score ( ¯ g ( x ) ), contribute to the learning of the feature model.
Revised: The auxiliary loss L aux ensures that all samples, regardless of their selective score ( ¯ g ( x ) ), contribute to the learning of the feature model.

Original: λ
Revised: λ

Original: and α are hyper-parameters controlling the trade-off of different terms.
Revised: and α are hyper-parameters controlling the trade-off of different terms.

Original: Unlike Deep Gamblers and Self-Adaptive Training, SelectiveNet trains a separate model for each target coverage c target .
Revised: Unlike Deep Gamblers and Self-Adaptive Training, SelectiveNet trains a separate model for each target coverage c target .

Original: In the SelectiveNet paper (Geifman & El-Yaniv, 2019), it has been suggested that the best performance is achieved when the training target coverage is equal to that of the evaluation coverage.
Revised: In the SelectiveNet paper (Geifman & El-Yaniv, 2019), it has been suggested that the best performance is achieved when the training target coverage is equal to that of the evaluation coverage.

Original: 3. A PPROACH : L EARN TO A BSTAIN
Revised: 3. A PPROACH : L EARN TO A BSTAIN

Original: Self-Adaptive Training (Huang et al., 2020) and Deep Gamblers (Ziyin et al., 2019) propose to tackle the selective classification problem by introducing a ( C + 1) -th class logit where the extra class logit represents abstention.
Revised: Self-Adaptive Training (Huang et al., 2020) and Deep Gamblers (Ziyin et al., 2019) propose to tackle the selective classification problem by introducing a ( C + 1) -th class logit where the extra class logit represents abstention.

Original: Let p θ ( ·| x ) represent the prediction network with softmax as the last layer.
Revised: Let p θ ( ·| x ) represent the prediction network with softmax as the last layer.

Original: This family of methods abstain if p θ ( C + 1 | x ) is above a threshold.
Revised: This family of methods abstain if p θ ( C + 1 | x ) is above a threshold.

Original: Here, we unify the notations of these abstention and selection methods under the same Selective Classification framework (See
Revised: Here, we unify the notations of these abstention and selection methods under the same Selective Classification framework (See

Original: Section 3.1) with the following soft selection function: ¯ g ( x ) = 1 − p θ ( C + 1 | x ) .
Revised: Section 3.1) with the following soft selection function: ¯ g ( x ) = 1 − p θ ( C + 1 | x ) .

Original: Due to the space limitation, the formulation for Deep Gamblers is included in the Appendix.
Revised: Due to the space limitation, the formulation for Deep Gamblers is included in the Appendix.

Original: 3.3. S ELF -A DAPTIVE T RAINING
Revised: 3.3. S ELF -A DAPTIVE T RAINING

Original: In addition to learning a logit that represents abstention, Self-Adaptive Training (Huang et al., 2020) proposes to use a convex combination of labels and predictions as a dynamically moving training target instead of the fixed labels.
Revised: In addition to learning a logit that represents abstention, Self-Adaptive Training (Huang et al., 2020) proposes to use a convex combination of labels and predictions as a dynamically moving training target instead of the fixed labels.

Original: Let y i be the one-hot encoded vector representing of the label for a datapoint ( x i , y i ) where y i is the label.
Revised: Let y i be the one-hot encoded vector representing of the label for a datapoint ( x i , y i ) where y i is the label.

Original: Initially, the model is trained with a cross-entropy loss for a series of pre-training steps.
Revised: Initially, the model is trained with a cross-entropy loss for a series of pre-training steps.

Original: Afterwards, the model is updated according to a dynamically moving training target.
Revised: Afterwards, the model is updated according to a dynamically moving training target.

Original: The training target t i is initially set equal to the label t i ← y i such that the training target is updated according to t i ← α × t i + (1 − α ) × p
Revised: The training target t i is initially set equal to the label t i ← y i such that the training target is updated according to t i ← α × t i + (1 − α ) × p

Original: θ ( ·| x i ) s.t. α ∈ (0 , 1) after each model update.
Revised: θ ( ·| x i ) s.t. α ∈ (0 , 1) after each model update.

Original: Similar to Deep Gamblers, the model is trained to optimise a loss function that allows the model to also choose to abstain on hard samples instead of making a prediction:
Revised: Similar to Deep Gamblers, the model is trained to optimise a loss function that allows the model to also choose to abstain on hard samples instead of making a prediction:

Original: [Equation]
Revised: [Equation]

Original: where m is the number of datapoints in the batch.
Revised: where m is the number of datapoints in the batch.

Original: As training progresses, t i approaches p θ ( ·| x i ) .
Revised: As training progresses, t i approaches p θ ( ·| x i ) .

Original: The first term is similar to the Cross Entropy Loss and encourages the model to learn a good classifier.
Revised: The first term is similar to the Cross Entropy Loss and encourages the model to learn a good classifier.

Original: The second term encourages the model to abstain from making predictions for samples that the model is uncertain about.
Revised: The second term encourages the model to abstain from making predictions for samples that the model is uncertain about.

Original: This use of dynamically moving training target t i allows the model to avoid fitting on difficult samples as the training progresses.
Revised: This use of dynamically moving training target t i allows the model to avoid fitting on difficult samples as the training progresses.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: We motivate an alternative selection mechanism Softmax Response for Selective Classification models.
Revised: We motivate an alternative selection mechanism Softmax Response for Selective Classification models.

Original: We explain how the known state-of-the-art selective methods can be equipped with the proposed selection mechanism and why it further improves performance.
Revised: We explain how the known state-of-the-art selective methods can be equipped with the proposed selection mechanism and why it further improves performance.

Original: Inspired by semi-supervised learning, we also introduce an entropy-regularized loss function.
Revised: Inspired by semi-supervised learning, we also introduce an entropy-regularized loss function.

Original: 4. M OTIVATION
Revised: 4. M OTIVATION

Original: Recent state-of-the-art methods have proposed to learn selective models with architecture modifications such as an external logit/head.
Revised: Recent state-of-the-art methods have proposed to learn selective models with architecture modifications such as an external logit/head.

Original: These architecture modifications, however, act as regularization mechanisms that allow the method to train more generalizable classifiers (see Table 1).
Revised: These architecture modifications, however, act as regularization mechanisms that allow the method to train more generalizable classifiers (see Table 1).

Original: As a result, the claimed improved results from these models could actually be attributed to their classifiers being more generalizable.
Revised: As a result, the claimed improved results from these models could actually be attributed to their classifiers being more generalizable.

Original: For these selective models to have strong performance in selective classification they require the external logit/head to generalise in these sense that the external logit/head must select samples for which the classifier is confident of its prediction.
Revised: For these selective models to have strong performance in selective classification they require the external logit/head to generalise in these sense that the external logit/head must select samples for which the classifier is confident of its prediction.

Original: Since the logit/head has its own set of learned model parameters, this adds another potential mode of failure for a selective model.
Revised: Since the logit/head has its own set of learned model parameters, this adds another potential mode of failure for a selective model.

Original: Specifically, the learned parameters can fail to generalise and the logit/head may (1) suggest samples for which the classifier is not confident about and (2) reject samples for which the classifier is confident about.
Revised: Specifically, the learned parameters can fail to generalise and the logit/head may (1) suggest samples for which the classifier is not confident about and (2) reject samples for which the classifier is confident about.

Original: In the appendix (See Figure 5 and 6), we include examples of images that fail due to this introduced mode of failure.
Revised: In the appendix (See Figure 4 and 5), we include examples of images that fail due to this introduced mode of failure.

Original: As such, we propose that selective mechanisms should stem from the classifier itself instead of an external logit/head, avoiding this extra mode of failure.
Revised: As such, we propose that selective mechanisms should stem from the classifier itself instead of an external logit/head, avoiding this extra mode of failure.

Original: 4. S ELECTING ACCORDING TO THE CLASSIFIER
Revised: 4. S ELECTING ACCORDING TO THE CLASSIFIER

Original: The cross entropy loss function is a popular loss function for classification due to its differentiability.
Revised: The cross entropy loss function is a popular loss function for classification due to its differentiability.

Original: However, during evaluation, the most utilized metric is accuracy, i.e. , whether a datapoint is predicted
Revised: However, during evaluation, the most utilized metric is accuracy, i.e. , whether a datapoint is predicted

Original: [Table]
Revised: [Table]

Original: correctly.
Revised: correctly.

Original: In the cross-entropy objective of the conventional classification settings, p ( c | x i ) is a one-hot encoded vector; therefore, the the cross-entropy loss can be simplified as CE ( p ( ·| x i ) , p θ ( ·| x i ))
Revised: In the cross-entropy objective of the conventional classification settings, p ( c | x i ) is a one-hot encoded vector; therefore, the the cross-entropy loss can be simplified as CE ( p ( ·| x i ) , p θ ( ·| x i ))

Original: = − (cid:80) Cu =1 p ( u | x i ) log p θ ( u | x i )
Revised: = − (cid:80) Cu =1 p ( u | x i ) log p θ ( u | x i )

Original: =
Revised: =

Original: − log p θ ( y i | x i ) , i.e. , during optimization, the logit of the correct class is maximised.
Revised: − log p θ ( y i | x i ) , i.e. , during optimization, the logit of the correct class is maximised.

Original: Accordingly, the maximum value of logits can be interpreted as the model’s relative confidence of its prediction.
Revised: Accordingly, the maximum value of logits can be interpreted as the model’s relative confidence of its prediction.

Original: Therefore, a simple selection mechanism for a model would be to select according to the maximum predictive class score, ¯ g ( x ) = max u ∈{ 1 ,...C } p θ ( u | x i ) (aka
Revised: Therefore, a simple selection mechanism for a model would be to select according to the maximum predictive class score, ¯ g ( x ) = max u ∈{ 1 ,...C } p θ ( u | x i ) (aka

Original: Softmax Response (Geifman & El-Yaniv, 2017)).
Revised: Softmax Response (Geifman & El-Yaniv, 2017)).

Original: Alternatively, a model can also select according to its predictive entropy ¯ g ( x ) =
Revised: Alternatively, a model can also select according to its predictive entropy ¯ g ( x ) =

Original: − H ( p θ ( ·| x )) , a metric of the model’s uncertainty.
Revised: − H ( p θ ( ·| x )) , a metric of the model’s uncertainty.

Original: An in-depth discussion is included in in the Appendix B.
Revised: An in-depth discussion is included in in the Appendix B.

Original: 4. R ECIPE FOR B ETTER S ELECTIVE C LASSIFICATION
Revised: 4. R ECIPE FOR B ETTER S ELECTIVE C LASSIFICATION

Original: The recipe that we are providing for better selective classification is as follows:
Revised: The recipe that we are providing for better selective classification is as follows:

Original: Train a selective classifier ( e
Revised: Train a selective classifier ( e

Original: g. , SelectiveNet, Self-Adaptive Training, or Deep Gamblers).
Revised: g. , SelectiveNet, Self-Adaptive Training, or Deep Gamblers).

Original: Discard its selection mechanism: • For SelectiveNet: Ignore the selection head • For Self-Adaptive Training and Deep Gambler: Ignore the additional abstain logit and compute the final layer’s softmax on the original C class logits.
Revised: Discard its selection mechanism: • For SelectiveNet: Ignore the selection head • For Self-Adaptive Training and Deep Gambler: Ignore the additional abstain logit and compute the final layer’s softmax on the original C class logits.

Original: Use a classifier-based selection mechanism (e.g., Softmax Response) to rank the samples.
Revised: Use a classifier-based selection mechanism (e.g., Softmax Response) to rank the samples.

Original: Calculate the threshold value τ , based on the validation set, to achieve the desired target coverage and select samples with max logit greater than τ .
Revised: Calculate the threshold value τ , based on the validation set, to achieve the desired target coverage and select samples with max logit greater than τ .

Original: Empirically, we showin Section 5 that selecting via entropy or Softmax Response both outperform selecting according to the external head/logit.
Revised: Empirically, we show that selecting via entropy or Softmax Response both outperform selecting according to the external head/logit.

Original: From these results, we can conclude that the strong performance of these recent state-of-the-art methods were due to learning a more generalizable classifier rather than their proposed selection mechanisms.
Revised: From these results, we can conclude that the strong performance of these recent state-of-the-art methods were due to learning a more generalizable classifier rather than their proposed selection mechanisms.

Original: In Step 3, we experimented with both an entropy-based selection mechanism and Softmax Response but we found that Softmax Response performed better.
Revised: In Step 3, we experimented with both an entropy-based selection mechanism and Softmax Response but we found that Softmax Response performed better.

Original: Notably, Softmax Response does not require retraining and can be immediately applied to already deployed models for significant performance improvement at negligible cost.
Revised: Notably, Softmax Response does not require retraining and can be immediately applied to already deployed models for significant performance improvement at negligible cost.

Original: 4. E NTROPY - REGULARIZED L OSS F UNCTION
Revised: 4. E NTROPY - REGULARIZED L OSS F UNCTION

Original: Here, we highlight a similarity between semi-supervised learning and selective classification, which to our knowledge has not been explored before.
Revised: Here, we highlight a similarity between semi-supervised learning and selective classification, which to our knowledge has not been explored before.

Original: In the semi-supervised learning setting the training dataset consists of labelled and unlabelled data.
Revised: In the semi-supervised learning setting the training dataset consists of labelled and unlabelled data.

Original: A simple approach is to train solely on the labelled data and ignore the unlabelled data, i.e., training a model via supervised learning to the labelled data.
Revised: A simple approach is to train solely on the labelled data and ignore the unlabelled data, i.e., training a model via supervised learning to the labelled data.

Original: This is equivalent to having a weight of 1 for the labelled samples and 0 for the unlabelled samples.
Revised: This is equivalent to having a weight of 1 for the labelled samples and 0 for the unlabelled samples.

Original: However, this is suboptimal because it does not use any information from the unlabelled samples.
Revised: However, this is suboptimal because it does not use any information from the unlabelled samples.

Original: Similarly, in Selective Classification, samples that are selected tend to have a high weight close to(see, for example, in Section 3.2.1, the ¯ g ( x ) term in the objective) and samples that are not selected have a low weight close to 0.
Revised: Similarly, in Selective Classification, samples that are selected tend to have a high weight close to(see, for example, in Section 3.2.1, the ¯ g ( x ) term in the objective) and samples that are not selected have a low weight close to 0.

Original: One way that semi-supervised learning have proposed to tackle this is via an entropy minimization term.
Revised: One way that semi-supervised learning have proposed to tackle this is via an entropy minimization term.

Original: Entropy minimization is one of the most standard, well-studied, and intuitive methods for semisupervised learning.
Revised: Entropy minimization is one of the most standard, well-studied, and intuitive methods for semisupervised learning.

Original: It uses the information of all the samples and increases the model’s confidence in its predictions, including on the unlabelled samples, resulting in a better classifier.
Revised: It uses the information of all the samples and increases the model’s confidence in its predictions, including on the unlabelled samples, resulting in a better classifier.

Original: Inspired by the similarity in the objective of selective classification and the setting of semi-supervised learning, we propose an entropy-minimisation term for the objective function of selective classification methods:
Revised: Inspired by the similarity in the objective of selective classification and the setting of semi-supervised learning, we propose an entropy-minimisation term for the objective function of selective classification methods:

Original: [Equation]
Revised: [Equation]

Original:  where β is a hyperparameter that controls the impact.
Revised: ) where β is a hyperparameter that controls the impact.

Original: In our experiments, we found β = 0 .
Revised: In our experiments, we found β = 0 .

Original: 01 to perform well in practice.
Revised: 01 to perform well in practice.

Original: The entropy minimization term encourages the model to be more confident in its predictions, i.e. , increasing the confidence of the predicted class and decreasing the predictive entropy during training.
Revised: The entropy minimization term encourages the model to be more confident in its predictions, i.e. , increasing the confidence of the predicted class and decreasing the predictive entropy during training.

Original: Thus, it allows for better disambiguation between sample predictions.
Revised: Thus, it allows for better disambiguation between sample predictions.

Original: The larger coefficient on the cross-entropy term compared to that of the entropy-minimization term ensures that increasing the confidence of correct predictions are prioritised, benefitting Softmax Response.
Revised: The larger coefficient on the cross-entropy term compared to that of the entropy-minimization term ensures that increasing the confidence of correct predictions are prioritised, benefitting Softmax Response.

Original: In Section 5, we show that this proposed loss function based on semi-supervised learning improves the performance in Selective Classification by a significant margin.
Revised: In Section 5, we show that this proposed loss function based on semi-supervised learning improves the performance in Selective Classification by a significant margin.

Original: These results opens the door to future exploration of the connection between Selective Classification and semi-supervised learning.
Revised: These results opens the door to future exploration of the connection between Selective Classification and semi-supervised learning.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: For the following experiments, we evaluate the following state-of-the-art methods (1) SelectiveNet (SN), (2) Self-Adaptive Training (SAT), and (3) Deep Gamblers.
Revised: For the following experiments, we evaluate the following state-of-the-art methods (1) SelectiveNet (SN), (2) Self-Adaptive Training (SAT), and (3) Deep Gamblers.

Original: Furthermore, we compare the performance of these methods with various selection mechanisms (1) original selection mechanism, (2) Ent.:entropy, and (3) SR: Softmax Response (our proposed method).
Revised: Furthermore, we compare the performance of these methods with the following selection mechanisms (1) original selection mechanism and (2) SR: Softmax Response (our proposed method).

Original: Due to space limitations, the table results forDeep Gamblers and a vanilla classifier are included in the Appendix.
Revised: Due to space limitations, the table results for a vanilla classifier is included in the Appendix with several additional results.

Original: The goal of our experimental evaluation is to answer the following questions: (1) Is the superior performance of recent state-of-the-art methods due to their proposed external head/logit selection mechanisms?
Revised: The goal of our experimental evaluation is to answer the following questions: (1) Is the superior performance of recent state-of-the-art methods due to their proposed external head/logit selection mechanisms?

Original: (2) What is the state-of-the-art selection mechanism?(3) How scalable are the selective methods for larger datasets?
Revised: Taking this further, what is the state-of-the-art selection mechanism?

Original: (4) Does the proposed entropy-regularized loss function improve Selective Classification results?
Revised: (2) Does the proposed entropy-regularized loss function improve the effectiveness of Softmax Response for

Original: (5) Given the implications of our extensive analysis and the surprising results to questions (1) to (4), what is the new state-of-the-art method for Selective Classification? 
Revised: Selective Classification? (3) What is the new state-of-the-art method for Selective Classification? How scalable are selective methods for larger datasets with larger number of classes?

Original: 5. D ATASETS
Revised: 5. D ATASETS

Original: We introducetwo new datasets, Imagenet100 and ImagenetSubset, for the selective classification problem setting and benchmark the existing state-of-the-art methods.
Revised: We introduce new datasets: StanfordCars, Food101, Imagenet, Imagenet100 and ImagenetSubset, for the selective classification problem setting and benchmark the existing state-of-the-art methods.

Original: We propose Imagenet100 asa realistic non-saturated dataset that can be evaluated at a wide range of coverages ( 10 − 100% ).
Revised: We propose StanfordCars, Food101, Imagenet, and Imagenet100, as realistic non-saturated datasets that can be evaluated at a wide range of coverages ( 10 − 100% ).

Original: In addition, we propose ImagenetSubset as a collection of datasets to evaluate the scalability of the methods for different number of classes.
Revised: In addition, we propose ImagenetSubset as a collection of datasets to evaluate the scalability of the methods for different number of classes.

Original: This is in contrast to the existing Selective Classification research which mainly have focused on small datasets such as CIFAR-10 with 10 or less classes, low resolution images (64x64 or less), and  high coverages ( 70%+) .
Revised: This is in contrast to the existing Selective Classification research which mainly have focused on small datasets such as CIFAR-10 with 10 or less classes, low resolution images (64x64 or less), and very low error (The error at 80% coverage is already lower than 1%) so this dataset is limited to high coverages ( 70%+) .

Original: The results of the previously introduced datasets indicate saturation, e.g. , 99 .
Revised: The results of the previously introduced datasets indicate saturation, e.g. , 0 .

Original: 7% accuracy atthe 70% coverage, discouraging experiments with lower coverages, which, in turn, robs researchers of achieving conclusive results.
Revised: 3% error at 70% coverage, discouraging experiments with lower coverages, which, in turn, prevents researchers from achieving conclusive results.

Original: CIFAR-10.
Revised: Imagenet/Imagenet100/ImagenetSubset.

Original: (Krizhevsky, 2009)The CIFAR-10 dataset comprises of small images: 50,000 images for training and10,00 images for evaluation  split into 10 classes.Each image is of size 32 × 32 × 3 . In our experiments, the selective classifiers are evaluated on coverages ranging from 70% to 100% .
Revised: Imagenet (Deng et al., 2009) comprises of 1,300 images per class and evaluation data comprising of 50,000 images split into 1,000 classes.

Original: Imagenet100. We propose to evaluate methods on Imagenet100 (Tian et al., 2020). The dataset was originally proposed for contrastive learning, but here we propose Imagenet100 as a standard benchmark for selective classification. The training data comprises ofapproximately 1,300 images per class and evaluation data comprises 5,000 images split into 100 classes.
Revised: Imagenet100 (Tian et al., 2020) is a subset of Imagenet which comprising of 100 classes.

Original: In our experiments, we evaluate classifiers on coverages between 10% and 100% .

Original: ImagenetSubset. We create a collection of datasets with varying number of classes (difficulty) from 25 to 175 in increments of 25.
Revised: ImagenetSubset is created as a collection of datasets with varying number of classes (difficulty) from 25 to 175 in increments of 25.

Original: The classes are sampled randomly such that datasets with less classes are subsets of those with more classes.
Revised: The classes are sampled randomly such that datasets with less classes are subsets of those with more classes.

Original: The complete list of selected classes in each dataset subset is available in the Appendix 1 .
Revised: The complete list of selected classes in each dataset subset is available in the Appendix 1 .

Original: These datasets help us evaluate the models’ performance with respect to the difficulty (scalability) of the task.
Revised: ImagenetSubset evaluates the models’ performance with respect to the difficulty (scalability) of the task.

Original: Similar to Imagenet100, the trainingdata comprises of approximately 1,300 imagesper class, and the evaluation data comprises 50 images per class.
Revised: Food101. The Food dataset (Bossard et al., 2014) contains 75750 training images and 25250 testing images split into 101 food categories.

Original: SelectiveNet
Revised: StanfordCars. The Cars dataset (Krause et al., 2013) contains 8,144 training images and 8, testing images split into 196 classes of cars. Unlike prior works which typically evaluate StanfordCars for transfer learning, in this work, the models are trained from scratch.

Original: [Table]
Revised: [Table]

Original: the bolded result identifies the lowest (best) selective risk and underlined result identifies the result with the second lowest (second best) selective risk. Selective Risk corresponds to the predictive error % for the selected samples.
Revised: CIFAR-10. The CIFAR-10 dataset (Krizhevsky, 2009) comprises of small images: 50,000 images for training and 10,00 images for evaluation split into 10 classes. Each image is of size 32 × 32 × 3 .

Original: 5. E XPERIMENT D ETAILS
Revised: 5. E XPERIMENT D ETAILS

Original: For our experiments, we used the publicly available official implementations of Deep Gamblers and Self-Adaptive Training.
Revised: For our experiments, we adapted the publicly available official implementations of Deep Gamblers and Self-Adaptive Training 2 .

Original: Experiments on SelectiveNet were conducted with our Pytorch implementation of the method which closely follows the details provided in the original paper (Geifman &
Revised: Experiments on SelectiveNet were conducted with our Pytorch implementation of the method which follow the details provided in the original paper (Geifman &

Original: El-Yaniv, 2019) .
Revised: El-Yaniv, 2019).

Original: For the CIFAR-10 experiment results to remain comparable, we follow the experimental details proposed in the original papers of SelectiveNet (Geifman & El-Yaniv, 2019), Deep Gamblers (Huang et al., 2020), and Self-Adaptive Training (Ziyin et al., 2019).

Original: Specifically, a VGG16 architecture is used.

Original: For the Imagenet100 and ImagenetSubset datasets, we use a ResNet34 architecture for Deep Gamblers, Self-Adaptive Training, and the main body block ofSelectiveNet.
Revised: For the StanfordCars, Food101, Imagenet100, and ImagenetSubset datasets, we use a ResNet34 architecture for Deep Gamblers, Self-Adaptive Training, and the main body block of

Original: The models are trained for 500 epochs using ADAM (Kingma & Ba, 2014) with a mini-batch size of 64. The learning rate was reduced by 0.5 every 25 epochs.
Revised: SelectiveNet.

Original: We follow the original papers with regards to every other hyperparameter.
Revised: Following prior work, we use a VGG16 architecture for the CIFAR-10 experiments.

Original: In the experiments involving theproposed entropy minimization loss function, the hyperparameter β that controls the weight of the penalty is set to 0 .
Revised: We tuned the entropy minimization loss function hyperparameter with the following values: β ∈ { 0 . 1 , 0 . 01 , 0 .

Original: All CIFAR-10 experiments were run withseeds and all Imagenet-related experiments were run with 3 seeds.
Revised: CIFAR10, Food101, and StanfordCars experiments were run with 5 seeds. Imagenet-related experiments were run with 3 seeds.

Original: Details regarding hyperparameter tuning are included in the Appendix.
Revised: Additional details regarding hyperparameters are included in the Appendix.

Original: 5. R ESULTS 5.3. CIFAR-
Revised: 5. R ESULTS 5.3. C ORRECTING T HE M ISCONCEPTION ABOUT THE SELECTION MECHANISM

Original: We emphasize that we are particularly interested in the relative selective performance difference between the various selection methods. In Table 2, we are comparing the various selection mechanisms ( i.e. , original, Ent., and SR) on a given Selective Classification method ( i.e. , SelectiveNet and SAT).
Revised: In Table 2, we compare the different selection mechanisms for a given selective classification method (SelectiveNet, Deep Gamblers, and Self-Adaptive Training). The results show that for each of these trained selective classifiers, their original selection mechanisms are suboptimal; in fact, selecting via

Original: In our experiments, we found that Self-Adaptive Training with Softmax Response and prediction entropy as the selection mechanisms on average outperform both SelectiveNet and SAT, and achieve state-of-the-art results. Even though Softmax Response and prediction entropy demonstrated superior performance on average, the difference in selective performance amongst methods was not significant. We attribute the lack of a clear significant difference in performance amongst methods to the saturatedness of the small CIFAR-10 dataset. This is supported by the results in the next subsections.
Revised: Softmax Response outperforms their original selection mechanism.

Original: In Table 3, we compare the results for different selective classification models with different selection mechanisms. We see that both the  selection mechanisms based on the classifier itself (predictive [Table]
Revised: These results suggest that (1) the strong performance of these methods were due to them learning a more generalizable model rather than their proposed external head/logit selection mechanisms and (2) the selection mechanism should stem from the classifier itself rather than a separate head/logit.

Original: [Figure] entropy and Softmax Response) significantly outperform the original selection mechanisms of the proposed methods.
Revised: We see that Softmax Response is the state-of-the-art selection mechanism.

Original: As a result, this suggests that (1) the strong performance of these methods were due to them learning a more generalizable model rather than their proposed external head/logit selection mechanisms and (2) the selection mechanism should stem from the classifier itself rather than a separate head/logit We see that Softmax Response is the state-of-the-art selection mechanism.
Revised: It is important to note that this performance gain is achieved by simply changing the selection mechanism of the pre-trained selective model without any additional computational cost. This observation applies to SN, DG, and SAT models.

Original: An interesting result  is that at low coverages ( 30% , 20% , and 10% ), SelectiveNet’s performance progressively gets worse.
Revised: An interesting result from this experiment is that at low coverages ( 30% , 20% , and 10% ), SelectiveNet’s performance progressively gets worse.

Original: We hypothesize that this is due to the optimisation process of SelectiveNet that allows the model to disregard ( i.e. , assign lower weight to their loss) a vast majority of samples during training at little cost, i.e. , ¯ g ( x ) ≈ 0 , especially when the target coverage is as low as 10% .
Revised: We hypothesize that this is due to the optimisation process of SelectiveNet that allows the model to disregard ( i.e. , assign lower weight to their loss) a vast majority of samples during training at little cost, i.e. , ¯ g ( x ) ≈ 0 , especially when the target coverage is as low as 10% .

Original: In contrast, Deep Gamblers and Self-Adaptive Training models are equally optimised over all samples regardless of their selection.
Revised: In contrast, Deep Gamblers and Self-Adaptive Training models are equally optimised over all samples regardless of their selection.

Original: In addition, we see that the Imagenet100 results are significantly less saturated than that of CIFAR-10.
Revised: [Table]

Original: On CIFAR-10, SelectiveNet achieves 99+% accuracy at 80% coverage. In contrast, on Imagenet100, SelectiveNet achieves and 94% at 80% coverage. In Table 5, we evaluate the efficacy of the proposed entropy-regularised loss function (Section 4.4) in training a selective classifier.
Revised: [Table]

Original: In this experiment, we focus on Self-Adaptive Training since it is the state-of-the-art method to the best of our knowledge.
Revised: In these experiments, we focus on Self-Adaptive Training as it is the state-of-the-art selective model.

Original: As demonstrated in Table  5, training with  the entropy minimization loss function converges to a more optimal model.
Revised: In Table 3, 4, and 5, we compare Self-Adaptive Training (SAT), SAT with SR (Softmax Response) selection mechanism, and SAT with SR and EM (Entropy-Minimization) on the Imagenet, StanfordCars, Food101, and CIFAR10 datasets.

Original: More importantly, the a clear and considerable improvement can be observed across all coverages when using Softmax Response, e.g. , 62 .
Revised: The results show that SAT+EM+SR achieves stateof-the-art performance across all coverages. For example, in StanfordCars, at 70% coverage, we see a raw 5 .

Original: 7% , and 58 .
Revised: 5% absolute improvement ( 25% relative reduction) in selective classification error by using our proposed method EM+SR. In Food101, at 70% coverage, we see a raw 1 .

Original: 5% relative improvement at 10% , 20% , 30% , and 40% coverages, respectively.
Revised: 37% absolute reduction ( 28% relative reduction) in selective classification error.

Original: Our results show that training Self-Adaptive Training with our proposed entropyregularised loss function and selecting according to Softmax Response achieves new state-of-the-art results by a very significant margin. 5.3. I MAGENET S UBSET
Revised: The clear and considerable improvement across all coverages when using Softmax Response selection mechanism rather than the original selection mechanism. These results further confirm our surprising finding that existing selection mechanisms are suboptimal.

Original: In this experiment (see Figure 1), we evaluate SelectiveNet and Self-Adaptive Training models on 70% coverage.
Revised: In the Appendix we further include (1) risk-coverage curves and (2) results for several network architectures.

Original: Consistent with the Imagenet100 experiments, we see that both the selection mechanisms based on the classifier itself (predictive entropy and Softmax Response) significantly outperform the original selection mechanisms of the proposed methods.
Revised: The results of those experiments show that our proposed methodology generalises across different network architectures. [Table] For the CIFAR-10 experiments (Table 5), the results for the different methods are within confidence intervals.

Original: These results further support our conclusion
Revised: Since the selective classification errors are very small, it is difficult to draw conclusions from such results.

Original: [Table]
Revised: On CIFAR-10, SAT achieves 99+% accuracy at 80% coverage. In contrast, on Imagenet100, SAT achieves 95% at 80% coverage. The saturation of CIFAR-10 is further highlighted in previous works which show improvements on the dataset (Geifman & El-Yaniv, 2019;

Revised: Ziyin et al., 2019; Huang et al., 2022) on the scale of 0 .

Revised: 1 − 0 .

Original: [Table]
Revised: [Table]

Original: that (1) the strong performance ofthese methods were due to them learning a more generalizable model and (2) the selection mechanism should stem from the classifier itself rather than a separate head/logit.
Revised: To evaluate the scalability of the proposed methodology with respect to the number of classes

Original: Similarly, we see that Softmax Response is the  state-of-the-art selection mechanism. In the experiments, we see that SelectiveNet struggles to scale to harder tasks.
Revised: [Table] we evaluate our proposed method SAT+EM+SR with the previous state-of-the-art SAT on ImagenetSubset. In Table

Original: Accordingly, the achieved improvement in selective accuracy with Softmax Response (SR) increases as the number of classes increase. This suggests that the proposed selection mechanism is more beneficial for SelectiveNet as the difficulty of the task increases, i.e. , improves scalability.
Revised: [Table]

Original: In Table 4, we see once again that Self-Adaptive Training with our proposed entropy-regularised loss function and selecting according to Softmax Response outperforms the previous state-of-the-art (vanilla Self-Adaptive Training) by a very significant margin (up to 85% relative improvement) across all sizes of datasets.
Revised: we see once again that Self-Adaptive Training with our proposed entropy-regularised loss function and selecting according to Softmax Response outperforms the previous state-of-the-art (vanilla Self-Adaptive Training) by a very significant margin (up to 85% relative improvement) across all sizes of datasets.

Original: Due to the space limitations, thetable results for Figure 1 and results for the other coverages of Table 4 are included in the Appendix.
Revised: Due to the space limitations, the results for the other coverages of Table 6 are included in the Appendix.

Revised: 5.3. E NTROPY -M INIMIZATION O NLY , S OFTMAX R ESPONSE S ELECTION O NLY , OR B OTH ?

Revised: In this experiment, we show that applying EM or SR alone provide gains.

Revised: However, to achieve state-of-the-art results by a large margin, it is crucial to use the combination of both SR and EM.

Revised: Table 4 shows that using only the entropy-minimization (SAT-EM) slightly improves the performance of

Original:   C ONCLUSION
Revised: SAT. However, SAT+EM+SR (SAT+EM in conjunction with SR selection mechanism) improves upon SAT+SR and SAT+EM significantly, achieving new state-of-the art results for selective classification.  C ONCLUSION

Original: In this work, we analysed the state-of-the-art Selective Classification methods and concluded that their strong performance is owed to learning a more generalisable classifier rather, yet their suggested selective solutions are suboptimal.
Revised: In this work, we analysed the state-of-the-art Selective Classification methods and concluded that their strong performance is owed to learning a more generalisable classifier rather, yet their suggested selective solutions are suboptimal.

Original: Accordingly, we showed that selection mechanisms based on the classifier itself outperform the state-of-the-art selection methods.
Revised: Accordingly, we showed that selection mechanisms based on the classifier itself outperforms the state-of-the-art selection methods.

Original: These results suggest that future work in selective classification should explore selection mechanisms based on the classifier itself rather than following recent works which proposed architecture modifications.
Revised: These results suggest that future work in selective classification should explore selection mechanisms based on the classifier itself rather than following recent works which proposed architecture modifications.

Original: We also highlighted a connection between selective classification and semi-supervised learning, which to our knowledge had not been explored.
Revised: Moreover, we also highlighted a connection between selective classification and semi-supervised learning, which to our knowledge has not been explored before.

Original: We show that a common technique in semi-supervised learning, namely, entropy-minimization, greatly improves performance in selective classification, opening the door to further exploration of the relationship between these two fields.
Revised: We show that a common technique in semi-supervised learning, namely, entropy-minimization, greatly improves performance in selective classification, opening the door to further exploration of the relationship between these two fields.

Original: From a practical perspective, we showed the selecting according to classification scores is the stateof-the-art selection mechanism for comparison.
Revised: From a practical perspective, we showed that selecting according to classification scores is the SOTA selection mechanism for comparison.

Original: Importantly, this method can be applied to an already deployed trained selective classification model and instantly improve performance at negligible cost.
Revised: Importantly, this method can be applied to an already deployed trained selective classification model and instantly improve performance at negligible cost.

Original: In addition, we showed a selective classifier trained with the entropy-regularised loss and with selection according to Softmax Response achieves new state-of-the-art results by a significant margin (up to 80% relative improvement).
Revised: In addition, we showed a selective classifier trained with the entropy-regularised loss and with selection according to Softmax Response achieves new state-of-the-art results by a significant margin.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: In our experiments we build on the official implementations of  Deep Gamblers (Link: https://github.com/Z-T-WANG/NIPS2019DeepGamblers) and Self-Adaptive Training (Link: https://github.com/LayneH/SAT-selective-cls).
Revised: In our experiments, we build on the official implementations of Self-Adaptive Training available at https://github.com/LayneH/SAT-selective-cls. Our code is available at https://github.com/BorealisAI/towards-better-sel-cls. The experiments with Deep Gamblers (Link: https://github.com/Z-T-WANG/NIPS2019DeepGamblers) are run using the official implementaiton.

Original: Our Pytorch implementation of SelectiveNet follows the details in the original paper.
Revised: Our Pytorch implementation of SelectiveNet follows the details in the original paper.

Original: The hyperparameters and implementation detailsfor the methods and datasets are available inthe Appendix C, Section and Section 5.
Revised: The implemen- tation details are available in Section 4. The hyperparameters are available in Section 5 and Appendix C. A CKNOWLEDGEMENTS

Original: The code will be released with the camera-ready version
Revised: The authors acknowledge funding from the Quebec government.

Original: A A PPENDIX : A DDITIONAL B ACKGROUND AND B ROADER I MPACT A. D EEP G AMBLERS
Revised: A A PPENDIX : A DDITIONAL B ACKGROUND AND B ROADER I MPACT A. D EEP G AMBLERS

Original: Inspired by portfolio theory, Deep Gamblers proposes to train the model using the following loss function:
Revised: Inspired by portfolio theory, Deep Gamblers proposes to train the model using the following loss function:

Original: [Equation]
Revised: [Equation]

Original: where m is the number of datapoints in the batch and o is a hyperparameter controlling the impact of the abstain logit.
Revised: where m is the number of datapoints in the batch and o is a hyperparameter controlling the impact of the abstain logit.

Original: Smaller values of o encourages the model to abstain more often.
Revised: Smaller values of o encourages the model to abstain more often.

Original: However, o ≤makes it ideal to abstain for all datapoints and o > C makes it ideal to predict for all datapoints.
Revised: However, o ≤makes it ideal to abstain for all datapoints and o > C makes it ideal to predict for all datapoints.

Original: As a result, o is restricted to be between 1 and C .
Revised: As a result, o is restricted to be between 1 and C .

Original: Note that the corresponding loss function with large values of o is approximately equivalent to the Cross Entropy loss.
Revised: Note that the corresponding loss function with large values of o is approximately equivalent to the Cross Entropy loss.

Original: A. B ROADER I MPACT
Revised: A. B ROADER I MPACT

Original: The broader impact of this work depends on the application of the selective model.
Revised: The broader impact of this work depends on the application of the selective model.

Original: In terms of the societal impact, fairness in selection remains a concern as lowering the coverage can magnify the difference in recall between groups and increase unfairness (Jones et al., 2021; Lee et al., 2021).
Revised: In terms of the societal impact, fairness in selection remains a concern as lowering the coverage can magnify the difference in recall between groups and increase unfairness (Jones et al., 2021; Lee et al., 2021).

Original: The calibration step performed on the validation set assumes the validation and test data are sampled from the same distribution.
Revised: The calibration step performed on the validation set assumes the validation and test data are sampled from the same distribution.

Original: Hence, in the case of out-of-distribution test data, a selective classifier calibrated to, for example,
Revised: Hence, in the case of out-of-distribution test data, a selective classifier calibrated to, for example,

Original: When evaluating, Selective Classifiers may choose to predict samples with easier classes more than hard to predict classes.
Revised: When evaluating, Selective Classifiers may choose to predict samples with easier classes more than hard to predict classes.

Original: Thus, it would be undesirable in fairness applications that require equal coverage amongst the different classes.
Revised: Thus, it would be undesirable in fairness applications that require equal coverage amongst the different classes.

Original: B A PPENDIX : A LTERNATIVE M OTIVATION
Revised: B A PPENDIX : A LTERNATIVE M OTIVATION

Original: In the selective classification problem setting, the objective is to select c target proportion of samples for prediction according to the value outputted by a selection function, ¯ g ( x ) .
Revised: In the selective classification problem setting, the objective is to select c target proportion of samples for prediction according to the value outputted by a selection function, ¯ g ( x ) .

Original: Since each datapoint ( x i , y i ) is an i.i.d.
Revised: Since each datapoint ( x i , y i ) is an i.i.d.

Original: sample, it is optimal to iteratively select from the dataset D the sample x ∗ that maximizes the selection function, i.e. , x ∗ ∈ argmax x ∈D ¯ g ( x ) .
Revised: sample, it is optimal to iteratively select from the dataset D the sample x ∗ that maximizes the selection function, i.e. , x ∗ ∈ argmax x ∈D ¯ g ( x ) .

Original: , until the target coverage c target proportion of the dataset is reached.
Revised: , until the target coverage c target proportion of the dataset is reached.

Original: In other words, to select c target proportion of samples (coverage =
Revised: In other words, to select c target proportion of samples (coverage =

Original: c target ), it is sufficient to define the criterion ¯ g and select a threshold τ such that exactly c target proportion of samples satisfy ¯ g ( x ) >
Revised: c target ), it is sufficient to define the criterion ¯ g and select a threshold τ such that exactly c target proportion of samples satisfy ¯ g ( x ) >

Original: τ .
Revised: τ .

Original: B. S ELECTING VIA P REDICTIVE E NTROPY
Revised: B. S ELECTING VIA P REDICTIVE E NTROPY

Original: At test time, given a dataset of datapoints D , if the labels were available, the optimal criterion to select the datapoint x ∈ D that minimise the loss function would be according to:
Revised: At test time, given a dataset of datapoints D , if the labels were available, the optimal criterion to select the datapoint x ∈ D that minimise the loss function would be according to:

Original: [Equation]
Revised: [Equation]

Original: However, at test time, the labels are unavailable.
Revised: However, at test time, the labels are unavailable.

Original: Instead, we can use the model’s belief over what the label is, i.e. , the learned approximation p θ ( ·| x ) ≈ p ( ·| x ) .
Revised: Instead, we can use the model’s belief over what the label is, i.e. , the learned approximation p θ ( ·| x ) ≈ p ( ·| x ) .

Original: We know CE ( p θ ( ·| x ) , p θ ( ·| x
Revised: We know CE ( p θ ( ·| x ) , p θ ( ·| x

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: H ( p θ ( ·| x )) where H is the entropy function.
Revised: H ( p θ ( ·| x )) where H is the entropy function.

Original: As such, we can select samples according to
Revised: As such, we can select samples according to

Original: [Equation]
Revised: [Equation]

Original: In other words, entropy is an approximation for the unknown loss function.
Revised: In other words, entropy is an approximation for the unknown loss function.

Original: Accordingly, with respect to the discussed selection framework (Section 3.1), the samples with the largest negative entropy value, i.e. , ¯ g ( x ) = − H ( p θ ( ·| x )) are best nominees for selection.
Revised: Accordingly, with respect to the discussed selection framework (Section 3.1), the samples with the largest negative entropy value, i.e. , ¯ g ( x ) = − H ( p θ ( ·| x )) are best nominees for selection.

Original: In Figure 2a, we show the distribution of entropy for a trained vanilla classifier, empirically showing entropy to be strongly inversely correlated with the model’s ability to correctly predict the labels.
Revised: In Figure 1a, we show the distribution of entropy for a trained vanilla classifier, empirically showing entropy to be strongly inversely correlated with the model’s ability to correctly predict the labels.

Original: As a result, entropy is a good selection mechanism.
Revised: As a result, entropy is a good selection mechanism.

Original: We include results on CIFAR-10 and Imagenet for a vanilla classifier in Table 7 and Table 8.
Revised: We include results on CIFAR-10 and Imagenet for a vanilla classifier in Table 8 and Table 9.

Original: [Figure]
Revised: [Figure]

Original: B. S ELECTING VIA M AXIMUM P REDICTIVE C LASS L OGIT (S OFTMAX R ESPONSE )
Revised: B. S ELECTING VIA M AXIMUM P REDICTIVE C LASS L OGIT (S OFTMAX R ESPONSE )

Original: Given a model with well-calibrated confidences (Guo et al., 2017; Minderer et al., 2021) an interpretation of p θ ( u | x ) is a probability estimate of the true correctness likelihood, i e., p θ ( u | x ) is the likelihood that u is the correct label of x .
Revised: Given a model with well-calibrated confidences (Guo et al., 2017; Minderer et al., 2021) an interpretation of p θ ( u | x ) is a probability estimate of the true correctness likelihood, i e., p θ ( u | x ) is the likelihood that u is the correct label of x .

Original: Let y i be the correct label for x i .
Revised: Let y i be the correct label for x i .

Original: For example, givensamples { x 1 , . . .
Revised: For example, givensamples { x 1 , . . .

Original: , x 100 } with p θ ( u | x i )
Revised: , x 100 } with p θ ( u | x i )

Original: = 0 .
Revised: = 0 .

Original: 8 , we would expect approximately 80% of the samples to have u as its label.
Revised: 8 , we would expect approximately 80% of the samples to have u as its label.

Original: As a result, p θ ( y i | x i ) is the model’s probability estimate that the correct label is y i .
Revised: As a result, p θ ( y i | x i ) is the model’s probability estimate that the correct label is y i .

Original: In classification, the probability that the calibrated model predicts a datapoint x correctly is equivalent to the value of the max class logit, i.e. , max u ∈{ 1 ,...C } p θ ( u | x ) .
Revised: In classification, the probability that the calibrated model predicts a datapoint x correctly is equivalent to the value of the max class logit, i.e. , max u ∈{ 1 ,...C } p θ ( u | x ) .

Original: Logically, the sample x i that should be selected for clasification is the sample the model’s most likely to predict the sample correctly, i.e. , i = argmax j (cid:0) max u ∈{ 1 ,...
Revised: Logically, the sample x i that should be selected for clasification is the sample the model’s most likely to predict the sample correctly, i.e. , i = argmax j (cid:0) max u ∈{ 1 ,...

Original: C } p θ ( u | x j ) (cid:1) .
Revised: C } p θ ( u | x j ) (cid:1) .

Original: This selection is equivalent to selecting according to the following soft selection function ¯ g ( x ) = max u ∈{ 1 ,...C } p θ ( u | x ) .
Revised: This selection is equivalent to selecting according to the following soft selection function ¯ g ( x ) = max u ∈{ 1 ,...C } p θ ( u | x ) .

Original: Simply put, this is equivalent to selecting according to the maximum predictive class logit (aka Softmax Response (Geifman &
Revised: Simply put, this is equivalent to selecting according to the maximum predictive class logit (aka Softmax Response (Geifman &

Original: El-Yaniv, 2017)).
Revised: El-Yaniv, 2017)).

Original: In practice, neural network models are not guaranteed to have well-calibrated confidences.
Revised: In practice, neural network models are not guaranteed to have well-calibrated confidences.

Original: In
Revised: In

Original: Selective Classification, however, we threshold according to τ and select samples above the threshold τ for classification, so we do not use the exact values of the confidence (max class logit).
Revised: Selective Classification, however, we threshold according to τ and select samples above the threshold τ for classification, so we do not use the exact values of the confidence (max class logit).

Original: As a result, we do not need the model to necessarily have well-calibrated confidences.
Revised: As a result, we do not need the model to necessarily have well-calibrated confidences.

Original: Instead, it suffices if samples with higher confidences (max class logit) have a higher likelihood of being correct.
Revised: Instead, it suffices if samples with higher confidences (max class logit) have a higher likelihood of being correct.

Original: In Figure 2b, we show the distribution of max class logit for a trained vanilla classifier, empirically showing larger max class logit to be strongly correlated with model’s ability to correctly predict the label.
Revised: In Figure 1b, we show the distribution of max class logit for a trained vanilla classifier, empirically showing larger max class logit to be strongly correlated with model’s ability to correctly predict the label.

Original: As a result, max class logit is a good selection mechanism.
Revised: As a result, max class logit is a good selection mechanism.

Original: We include results on CIFAR-10 and
Revised: We include results on CIFAR-10 and

Original: Imagenet100 for a vanilla classifier in Table 7 and Table 8.
Revised: Imagenet100 for a vanilla classifier in Table 8 and Table 9.

Original: B. R ECIPE FOR B ETTER S ELECTIVE C LASSIFICATION
Revised: B. R ECIPE FOR B ETTER S ELECTIVE C LASSIFICATION

Original: In this section, we further illustrate how SelectiveNet’s original selection mechanism is suboptimal.
Revised: In this section, we further illustrate how SelectiveNet’s original selection mechanism is suboptimal.

Original: The optimisation of SelectiveNet’s selective loss L selective (See Section 3.2.1) aims to learn a selection head (soft selection model)
Revised: The optimisation of SelectiveNet’s selective loss L selective (See Section 3.2.1) aims to learn a selection head (soft selection model)

Original: ¯ g that outputs a low selection value for inputs with large cross-entropy loss and high selection value for inputs with low cross-entropy loss.
Revised: ¯ g that outputs a low selection value for inputs with large cross-entropy loss and high selection value for inputs with low cross-entropy loss.

Original: At test time, good performance of SelectiveNet depends on the generalisation of both the prediction and selection heads.
Revised: At test time, good performance of SelectiveNet depends on the generalisation of both the prediction and selection heads.

Original: However, learned models can at times fail to generalise.
Revised: However, learned models can at times fail to generalise.

Original: In Figure 3 and Figure 4, we show the distribution of entropy and max class logit for selected and not-selected samples according to a
Revised: In Figure 2 and Figure 3, we show the distribution of entropy and max class logit for selected and not-selected samples according to a

Original: SelectiveNet model.
Revised: SelectiveNet model.

Original: In the plots, we see SelectiveNet’s original selection mechanism selects several samples with large entropy and low max class logit.
Revised: In the plots, we see SelectiveNet’s original selection mechanism selects several samples with large entropy and low max class logit.

Original: In Table 3, we see that the selection mechanisms based on entropy and max class logit outperforms the original selection mechanism.
Revised: In Table ?? , we see that the selection mechanisms based on entropy and max class logit outperforms the original selection mechanism.

Original: This comparison further supports our argument that the selection mechanism should be rooted in the objective function instead of a separately calculated score.
Revised: This comparison further supports our argument that the selection mechanism should be rooted in the objective function instead of a separately calculated score.

Original: C A PPENDIX : A DDITIONAL E XPERIMENTAL D ETAILS C. H YPERPARAMETERS
Revised: C A PPENDIX : A DDITIONAL E XPERIMENTAL D ETAILS C. H YPERPARAMETERS

Original: Following (Geifman & El-Yaniv, 2019), SelectiveNet was trained with a target coverage rate and evaluated on the same coverage rate.
Revised: Following (Geifman & El-Yaniv, 2019), SelectiveNet was trained with a target coverage rate and evaluated on the same coverage rate.

Original: As a result, there are different models for each experimental coverage rate.
Revised: As a result, there are different models for each experimental coverage rate.

Original: In contrast, target coverage does not play a role in the optimization process of Deep
Revised: In contrast, target coverage does not play a role in the optimization process of Deep

Original: Gamblers and Self-Adaptive Training, hence, the results for different experimental coverages are computed with the same models.
Revised: Gamblers and Self-Adaptive Training, hence, the results for different experimental coverages are computed with the same models.

Original: All CIFAR-10 experiments were performed with 5 seeds.
Revised: All CIFAR-10 experiments were performed with 5 seeds.

Original: All Imagenet-related experiments were performed with 3 seeds.
Revised: All Imagenet-related experiments were performed with 3 seeds.

Original: For hyperparameter tuning, we split Imagenet100’s training data into 80% training data and 20% validation data evenly across the different classes.
Revised: For hyperparameter tuning, we split Imagenet100’s training data into 80% training data and 20% validation data evenly across the different classes.

Original: We tested the following values for the entropy minimization coefficient β ∈ { 0 . 001 , 0 . 01 , 0 .1 } and concluded that β = 0 .performs best.
Revised: We tested the following values for the entropy minimization coefficient β ∈ { 0 . 1 , 0 . 01 , 0 .

Original: For the final evaluation, we trained the model on the entire training data.
Revised: For the final evaluation, we trained the model on the entire training data.

Original: C. R EPLICATING CIFAR-10 RESULTS
Revised: Self-Adaptive Training models are trained using SGD with an initial learning rate of 0 . 1 and a momentum of 0 . 9 . Food101/Imagenet100/ImagenetSubset. The models were trained for 500 epochs with a mini-batch size of 128. The learning rate was reduced by 0.5 every 25 epochs.

Revised: The entropy-minimization term was β = 0 .

Original: For the CIFAR-10 experiment, we attempt to replicate the previously published results.
Revised: CIFAR-10. The models were trained for 300 epochs with a mini-batch size of 64. The learning rate was reduced by 0.5 every 25 epochs. The entropy-minimization term was β = 0 .

Original: Based on our experiments, Self-Adaptive Training at coverages of 100, 90, 80, 75, and 70 performed slightly better than reported in the paper, however, it performed slightly worse than reported on coverages of and 85 (Table 2).
Revised: StanfordCars. The models were trained for 300 epochs with a mini-batch size of 64. The learning rate was reduced by 0.5 every 25 epochs. The entropy-minimization term was β = 0 .

Original: Similarly, SelectiveNet performed slightly better on coverages 100, 95, and 85 and slightly worse on coverages 90, 80, 75, and 70 than reported in the original paper (Table 2).
Revised: Imagenet. The models were trained for 150 epochs with a mini-batch size of 256. The learning rate was reduced by 0.5 every 10 epochs. The entropy-minimization term was β = 0 .

Original: C. C OMPUTE
Revised: C. C OMPUTE

Original: The experiments were primarily run on a GTX 1080 Ti.
Revised: The experiments were primarily run on a GTX 1080 Ti.

Original: The CIFAR10 experiments took 1.5 hours for Self-Adaptive Training and Deep Gamblers.
Revised: The CIFAR10 experiments took 1.5 hours for Self-Adaptive Training and Deep Gamblers.

Original: SelectiveNet experiments took 3 hours each.
Revised: SelectiveNet experiments took 3 hours each.

Original: The
Revised: The

Original: Imagenet100 experiments took 2 days for Self-Adaptive Training and Deep Gamblers.
Revised: Imagenet100 experiments took 2 days for Self-Adaptive Training and Deep Gamblers.

Original: SelectiveNet experiments took 2.75 days each.
Revised: SelectiveNet experiments took 2.75 days each.

Original: The ImagenetSubset experiments took 0.5-4.5 days each for
Revised: The ImagenetSubset experiments took 0.5-4.5 days each for

Original: Self-Adaptive Training and Deep Gamblers, depending on the number of classes.
Revised: Self-Adaptive Training and Deep Gamblers, depending on the number of classes.

Original: SelectiveNet experiments took 0.75-5.5 days each, depending on the number of classes.
Revised: SelectiveNet experiments took 0.75-5.5 days each, depending on the number of classes.

Original: C. I MAGENET S UBSET : C LASSES
Revised: C. I MAGENET S UBSET : C LASSES

Original: ImagenetSubset comprises of multiple datasets ranging from 25 to 175 classes in increments of 25 , i.e. , {D 25 , D 50 , D 75 , D 100 , D 125 , D 150 , D 175 } .
Revised: ImagenetSubset comprises of multiple datasets ranging from 25 to 175 classes in increments of 25 , i.e. , {D 25 , D 50 , D 75 , D 100 , D 125 , D 150 , D 175 } .

Original: Let C 25 , C 50 , . . .
Revised: Let C 25 , C 50 , . . .

Original: , C 175 represent the classes of the respective datasets.
Revised: , C 175 represent the classes of the respective datasets.

Original: The classes for ImagenetSubset are uniform randomly sampled from the classes of Imagenet such that the classes of the smaller datasets are subsets of the classes of the larger datasets, i.e. D 25 ⊂ D 50 ⊂ D 75 ⊂ · · · ⊂ D 175 and C 25 ⊂ C 50 . . .
Revised: The classes for ImagenetSubset are uniform randomly sampled from the classes of Imagenet such that the classes of the smaller datasets are subsets of the classes of the larger datasets, i.e. D 25 ⊂ D 50 ⊂ D 75 ⊂ · · · ⊂ D 175 and C 25 ⊂ C 50 . . .

Original: C 175 .
Revised: C 175 .

Original: The list of Imagenet classes in each dataset is included below for reproducibility.
Revised: The list of Imagenet classes in each dataset is included below for reproducibility.

Original: C.4. C 
Revised: C.3.

Original: [Table]
Revised: [Equation]

Original: n n

Original: [Table]
Revised: [Table]

Original:  n
Revised: n03483316 n02177972 n03814906 n01753488 n n01755581 n02264363 n03290653 n13133613 n03929660 n04040759 n02317335 n n02865351 n03134739 n02102177 n04192698 n02814533 n

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [-[Table] n02408429 n02025239 n n03041632 n02484975 n02097209 n
 n03476684 n04579145 n01739381 n n
 n02229544 n09288635 n-]
Revised: [-[Table] n02408429 n02025239 n n03041632 n02484975 n02097209 n
 n03476684 n04579145 n01739381 n n
 n02229544 n09288635 n-]C.3.

Original: [Table]
Revised: [Table]

Original: n07583066 n
Revised: n

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Table]
Revised: [Table]

Original: [-n03450230 n01616318 n01641577 n03630383 n n02102973 n
 n02134084 n01729322 n03250847 n02099849 n03544143 n n03777754 n
 n01770081 n03255030 n01910747 n03016953 n03485407 n n-]
Revised: [-n03450230 n01616318 n01641577 n03630383 n n02102973 n
 n02134084 n01729322 n03250847 n02099849 n03544143 n n03777754 n
 n01770081 n03255030 n01910747 n03016953 n03485407 n n-][Table]

Original: [Table]
Revised: [Table]

Original: n02128757 n03763968 n01677366 n03483316 n02177972 n03814906 n01753488 n n01755581 n02264363 n03290653 n13133613 n03929660 n04040759 n02317335 n n02865351 n03134739 n02102177 n04192698 n02814533 n04090263 n01818515 n n  
Revised: n02109525 n03938244 n02123045 n04548362 n04612504 n04264628 n02108551 n n C.3.

Original: [Table]
Revised: [Table]

Original: n03930313 n02422106 n07714571 n02111277 n03706229 n03729826 n03344393 n n02090379 n06596364 n03187595 n04317175 n11939491 n04277352 n01807496 n n02093991 n09428293 n03207941 n02132136 n
Revised: n03724870 n02087046 n09421951 n02799071 n07717410 n02906734 n02206856 n n01740131 n04523525 n03496892 n04116512 n03743016 n03759954 n04462240 n n02137549 n03866082 n

Original: [Table]
Revised: [Table]

Original: n03530642 n[Table] n13044778 n03188531 n01729977 n01980166 n02101388 n01629819 n01773157 n[Table]
Revised: n02445715 n02974003 n01924916 n n

Original: [Table]

Original: n01924916 n n

Original: D A PPENDIX : A DDITIONAL R ESULTS
Revised: D A PPENDIX : A DDITIONAL R ESULTS

Original: Briefly summarised, the additional interesting results found are as follows:
Revised: Briefly summarised, the additional interesting results found are as follows:

Original: (1) In low coverage settings, selecting based on Softmax Response and Entropy on a vanilla classifier trained via the cross entropy loss outperform both SelectiveNet and Deep Gamblers.
Revised: (1) In low coverage settings, selecting based on Softmax Response and Entropy on a vanilla classifier trained via the cross entropy loss outperform both SelectiveNet and Deep Gamblers.

Original: (2) SelectiveNet outperforms
Revised: (2) SelectiveNet outperforms

Original: Deep Gamblers on moderate coverages ( 60% , 50% , and 40% ) which is not in par with the previously reported results.
Revised: Deep Gamblers on moderate coverages ( 60% , 50% , and 40% ) which is not in par with the previously reported results.

Original: We attribute the interesting results to our work being the first to evaluate these methods on large datasets at a wide range of coverages.
Revised: We attribute the interesting results to our work being the first to evaluate these methods on large datasets at a wide range of coverages.

Original: Since previous works have mainly focused on toy datasets and high coverages ( 70 + % ), they failed to capture these patterns.
Revised: Since previous works have mainly focused on toy datasets and high coverages ( 70 + % ), they failed to capture these patterns.

Original: The main takeaway of these results, however, is that, across all the reported methods, selecting via Softmax Response is best.
Revised: The main takeaway of these results, however, is that, across all the reported methods, selecting via Softmax Response is best.

Original: D. A NALYSIS OF S ELECTED I MAGES : S ELECTIVE N ET
Revised: D. A NALYSIS OF S ELECTED I MAGES : S ELECTIVE N ET

Original: We include in Figure 5 and Figure 6 examples of images where the selection head of SelectiveNet fails to generalise.
Revised: We include in Figure 4 and Figure 5 examples of images where the selection head of SelectiveNet fails to generalise.

Original: D. C OMPARISON OF B ASELINES WITH MC-D ROPOUT
Revised: D. S ELECTION M ECHANISMS : MC-D ROPOUT

Original: In Table 6, we see that MC-Dropout performs worse than the existing state-of-the-art methods for
Revised: In Table 7, we see that MC-Dropout performs worse than the existing state-of-the-art methods for

Original: Selective Classification.
Revised: Selective Classification.

Revised: [Table]

Revised: [Table]

Original: D. S ELECTION M ECHANISMS : V ANILLA C LASSIFIER D.3. CIFAR- 
Revised: D. S ELECTION M ECHANISMS : V ANILLA C LASSIFIER D.3. CIFAR-

Original: In Table 7, the difference in performance between selecting according to entropy and selecting according to Softmax Response is not significant.
Revised: In Table [Table] the difference in performance between selecting according to entropy and selecting according to Softmax Response is not significant.

Original: We attribute this marginal difference to the saturatedness of the CIFAR-10 dataset.
Revised: We attribute this marginal difference to the saturatedness of the CIFAR-10 dataset.

Original: D.3. I MAGENET
Revised: D.3. I MAGENET

Original: In Table 8, we see that selecting according to Softmax Response clearly outperforms selecting according to entropy.
Revised: In Table [Table] we see that selecting according to Softmax Response clearly outperforms selecting according to entropy.

Original: We see that Softmax Response learns a less generalizeable clasifier (See performance on 100% coverage) than Self-Adaptive Training, Deep Gamblers, and SelectiveNet.[Table] [Table]
Revised: We see that Softmax Response learns a less generalizeable clasifier (See performance on 100% coverage) than Self-Adaptive Training, Deep Gamblers, and SelectiveNet.

Original: [Table]

Original: However, interestingly, we found that Softmax Response outperforms both Deep Gamblers and
Revised: However, interestingly, we found that Softmax Response outperforms both Deep Gamblers and

Original: SelectiveNet on low coverages ( 10% , 20% , 30% ).
Revised: SelectiveNet on low coverages ( 10% , 20% , 30% ).

Original: Previous works failed to capture this pattern due to lack of evaluation on larger datasets and lower coverages.
Revised: Previous works failed to capture this pattern due to lack of evaluation on larger datasets and lower coverages.

Original: D. S ELECTION M ECHANISMS : D EEP G AMBLERS
Revised: D. S ELECTION M ECHANISMS : D EEP G AMBLERS

Original: CIFAR-10.
Revised: CIFAR-10.

Original: In these results (Table 9), we see that the difference in performance between the various selection mechanisms is marginal.
Revised: In these results (Table 10), we see that the difference in performance between the various selection mechanisms is marginal.

Original: Similar to the CIFAR-10 results for SelectiveNet and Self-Adaptive

Original: Training, it is difficult to make conclusions from these results.
Revised: Due to the marginal difference between errors, it is difficult to make conclusions from these results.

Original: Imagenet100.
Revised: Imagenet100.

Original: In Table 9, we see that selecting according to Softmax Response  clearly outperforms the original selection mechanism.Furthermore, selecting according to entropy also outperforms the original selection mechanism.
Revised: In Table 10, we see that selecting according to Softmax Response and Entropy clearly outperforms the original selection mechanism.

Original: ImagenetSubset.
Revised: ImagenetSubset.

Original: In Table 10, similar to Imagenet100, we see a clear substantial improvement when using Softmax Response as the selection mechanism instead of the original selection mechanism.
Revised: In Table 11, similar to Imagenet100, we see a clear substantial improvement when using Softmax Response as the selection mechanism instead of the original selection mechanism.

Original: Furthermore, we see that Entropy also outperforms the original selection mechanism.
Revised: Furthermore, we see that Entropy also outperforms the original selection mechanism.

Original: D. E NTROPY -M INIMIZATION R ESULTS

Original: CIFAR-10.
Revised: [Table]

Original: In Table 11, we see that the results are saturated and the entropy-minimization term does not affect the results significantly.
Revised: [Table] D. S ELECTION M ECHANISMS : S ELF -A DAPTIVE T RAINING

Original: ImagenetSubset.
Revised: ImagenetSubset.

Original: In addition to the Imagenet100 experiments, we also evaluate Self-Adaptive
Revised: In addition to the Imagenet100 experiments, we also evaluate Self-Adaptive

Original: Training trained with the proposed entropy-regularised loss function on ImagenetSubset.
Revised: Training trained with the proposed entropy-regularised loss function on ImagenetSubset.

Original: In  Table [Table]
Revised: In Figure 6 (and Table 12 and Table 13)

Original: [Table]
Revised: [Table]

Original: 12 and Table 13, we see that Self-Adaptive Training with Entropy-Minimization further improves performance significantly on ImagenetSubset, achieving state-of-the-art results.

Original: In Figure 7 (and Table 12 and Table 13), we see that training with the entropy-regularised loss function improves the scalability of Self-Adaptive Training when selecting according to Softmax
Revised: we see that training with the entropy-regularised loss function improves the scalability of Self-Adaptive Training when selecting according to Softmax

Original: Response.
Revised: Response.

Revised: In Figure

Original: [Table]
Revised: [Table]

Revised: the results for SelectiveNet, Deep Gamblers

Original: [Figure]
Revised: [Table]

Original: [Figure]
Revised: and Self-Adaptive Training on 70% coverage. Consistent with previous experiments, we see that both the selection mechanisms based on the classifier itself (predictive entropy and Softmax Response) significantly outperform the original selection mechanisms of the proposed methods. These results further support our conclusion that (1) the strong performance of these methods were due to them learning a more generalizable model and (2) the selection mechanism should stem from the classifier itself rather than a separate head/logit. Similarly, we see that Softmax Response is the state-of-the-art selection mechanism.

Original: [Table]
Revised: In the experiments, we see that SelectiveNet struggles to scale to harder tasks. Accordingly, the achieved improvement in selective accuracy with Softmax Response (SR) increases as the number of classes increase. This suggests that the proposed selection mechanism is more beneficial for SelectiveNet as the difficulty of the task increases, i.e. , improves scalability.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: significant margin. D. A DDITIONAL D ATASETS
Revised: D. A BLATION : V ARYING A RCHITECTURE

Original: StanfordCars: The Cars dataset (Krause et al., 2013) contains 8,144 training images and 8, testing images split into 196 classes of cars. We ran the experiment for 300 epochs.
Revised: In these experiments

Original: Food101: The Food dataset (Bossard et al., 2014) contains 75750 training images and 25250 testing images split into 101 food categories. We trained the model for 500 epochs. D.6. S TANFORD C ARS
Revised: [Table]

Original: In these experiments, we show generalizability across architectures of our proposed entropyminimization and softmax response methodology. 
Revised: we show generalizability across architectures of our proposed entropyminimization and softmax response methodology. In Tables 14,

Original: In Tables 14, 15, and 16, we see that applying the entropy-minimization and softmax response methodology improves upon the state-of-the-art method’s performance significantly.
Revised: [Table] and [Table] we see that applying the entropy-minimization and Softmax Response methodology improves upon the state-of-the-art method’s performance significantly.

Original: D.6. F OOD
Revised: D. R ISK -C OVERAGE P LOTS

Original: In this experiment (Table 17, we once again see that applying the entropy-minimization and softmax response methodology improves upon the state-of-the-art method’s performance significantly.
Revised: Figure 7 shows the risk coverage plots for Imagenet100, Food101, and StanfordCars results . D. L EARNING C URVES P LOTS Figure 8 shows that SAT and SAT+EM models have converged on StanfordCars.

Original: [Table]
Revised: [Table]

Original: 0.69 ± 0.
Revised: 0.69 ± 0.

Original: 0.93 ± 0.
Revised: 0.93 ± 0.

Original: 0.49 ± 0.
Revised: 0.49 ± 0.

Original: [Table]
Revised: [Table]

Original: 3.03 ± 0.

Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 2.23 ± 0.
Revised: 2.23 ± 0.

Original: 1.71 ± 0.
Revised: 1.71 ± 0.

Original: [Table]
Revised: [Table]

Original: 2.32 ± 0.
Revised: 2.32 ± 0.

Original: 1.84 ± 0.
Revised: 1.84 ± 0.

Original: [Table]
Revised: [Table]

Original: 2.65 ± 0.
Revised: 2.65 ± 0.

Original: 1.81 ± 0.
Revised: 1.81 ± 0.

Original: [Table]
Revised: [Table]

Original: 2.60 ± 0.
Revised: 2.60 ± 0.

Original: 1.68 ± 0.
Revised: 1.68 ± 0.

Original: [Table]
Revised: [Table]

Original: 2.86 ± 0.
Revised: 2.86 ± 0.

Original: 1.47 ± 0.
Revised: 1.47 ± 0.

Original: 1.79 ± 0.
Revised: 1.79 ± 0.

Original: 1.14 ± 0.
Revised: 1.14 ± 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] [Table]
Revised: [Figure] [Figure]