Generalizing Consistent Multi-Class Classification with Rejection to be Compatible with Arbitrary Losses
Anonymous Author(s)
AffiliationAddress email
Abstract
Classification with rejection (CwR) refrains from making a prediction to avoid critical misclassification when encountering test samples that are difficult to classify.
Though previous methods for CwR have been provided with theoretical guarantees,they are only compatible with certain loss functions, making them not flexibleenough when the loss needs to be changed with the dataset in practice.
In this paper,we derive a novel formulation for CwR that can be equipped with arbitrary lossfunctions while maintaining the theoretical guarantees.
First, we show that K -class CwR is equivalent to a ( K +1) -class classification problem on the original datadistribution with an augmented class, and propose an empirical risk minimizationformulation to solve this problem with an estimation error bound.
Then, we find anecessary and sufficient condition for the learning consistency of the surrogates constructed on our proposed formulation equipped with any classification-calibratedmulti-class losses, where consistency means the surrogate risk minimization implies the target risk minimization for CwR. Finally, experiments on benchmarkdatasets validate the effectiveness of our proposed method.
Introduction
In risk-sensitive multi-class classification applications (e g., medical diagnosis, healthcare, autonomous driving, and product inspections [12, 21, 38]), misclassification can cause serious oreven fatal consequences.
To alleviate this issue, many studies have been conducted on classificationwith rejection (CwR) [10, 6, 54, 12, 13, 15, 21, 45, 42, 38, 8], which can abstain from making anunsure prediction to prevent such critical misclassification.
Most of the previous studies follow the framework that provides the reject option with a pre-definedcost c which is lower than the misclassification cost 1 .
Given cost c , the problem is further formulatedas a risk minimization problem that aims to minimize the expectation of the zero-one- c loss, i e., thezero-one- c risk.
With the risk minimization process, the obtained classifier can balance the cost ofrejection and prediction by choosing to incur a rejection cost c if the misclassification risk is high. Due to the discontinuous nature of the zero-one- c loss, recent works focused on finding its continuoussurrogates to make the optimization problem tractable. A basic requirement for surrogate lossesis the consistency [56, 7, 47, 41], i.e., the surrogate risk minimization implies the zero-one- c riskminimization. Moreover, compared with the traditional K -class classification task where decisionsare normally made from the index of the maximum coordinate of a K -dimensional scoring function,the design of decision criteria in the CwR task is more elusive due to the existence of a reject option.
[Figure]
By adopting different classification and rejection criteria, various surrogates of the zero-one- c losshave been proposed with consistency analyses [6, 54, 12, 13, 42, 38, 8].
Classical studies focused on developing confidence-based methods [6, 54, 42, 38], which use theoutputs of classifiers as confidence values and set a real-valued threshold as the rejection rule.
Representative methods [54, 38] used surrogates that depend on class-posterior possibility estimation(CPE) [43, 50], which is challenging when using deep models [23].
Though some of them [6, 42, 32,22] could avoid CPE, most of them applied the modification of non-differentiable hinge/ramp-likesurrogates, and their performance was only validated with linear models.
To avoid the use of the confidence threshold, Cortes et al.
[12] provided an upper bound of the zeroone- c loss as the surrogate that allows the use of a separated rejector and can be trained simultaneouslywith the classifier, which is regarded as classifier-rejector methods .
Though these methods achievedstate-of-the-art performance in binary classification scenarios, they only provided a consistencyguarantee for hinge-like and exponential losses and cannot be directly generalized to the multi-classscenario as shown in Ni et al.
Charoenphakdee et al.
[8] showed that K -class CwR can bedecomposed into K binary cost-sensitive classification problems [16, 44, 11] and proposed a familyof surrogates are the ensembles of arbitrary binary classification losses, which can avoid CPE and theuse of confidence threshold with properly chosen losses when the cost function is constant.
Mozannarand Sontag [35] provided a modified version of the cross entropy loss as the surrogate for the task oflearning to defer [35, 36] that can also be used in CwR, while its optimal solution still relies on CPE.
In summary, previous works only took limited types of losses into consideration, and there lacks atheoretically grounded framework that can cover all the surrogates used in multi-class classification.
In this paper, we propose a novel framework for CwR that allows the use of arbitrary surrogate lossesused in traditional multi-class classification as long as they are classification-calibrated, includingbut not limited to the well-known cross entropy loss, mean absolute error, focal loss [30, 9], and thepairwise/one-versus-all generalizations of binary margin losses [56].
Thanks to the flexible choices oflosses, we be free of the restricted analyses on the consistency of certain surrogates.
An overview ofour framework is shown in Figure 1.
We summarize the main contributions of this work as follows:
• We disclose the equivalence between K -class CwR and a ( K +1) -class
classification problemon the original data distribution with an augmented class, by showing the equality between theirclassification risks.•
We propose a formulation of surrogates for ℓ 01 c that can recover the surrogate risk of a ( K +1)
-class classification task only with the K -class training distribution, and derive an estimation errorbound for its empirical risk minimization.•
We find a necessary and sufficient condition for the consistency of the proposed family of surrogatesw.r.t.
the zero-one- c loss that allows the use of any calibrated multi-class surrogates.
We for the first time provide an analysis on the calibration of the generalized cross entropy loss [57] that benefits from both the cross entropy loss and mean absolute error, and experimentally demonstrate that it is suitable for our proposed framework.
Preliminaries
In this section, we provide preliminary knowledge of CwR and calibrated surrogate losses, anddiscuss the consistency in CwR.
2.1 Classification with Rejection
The problem setting of CwR is based on the cost-based framework [10].
Let us denote by X the feature space, Y = { 1 , 2 , . .
, K } the label space , and Y ® = { 1 , 2 , . . .
, K, ® } the label space with a reject option.
We are given instance-label pairs { ( x i , y i ) } ni =1 independently and identically drawn from an underlying distribution with probability density p ( x , y ) .
The goal of CwR is to train a classifier f : X → Y ® that can abstain from making a decision, where ® denotes the reject option.
The evaluation metric of this task is the zero-one- c loss ℓ 01 c , which can be expressed as a variant of the traditional zero-one loss ℓ 01 ( f ( x ) , y ) =
[[ f ( x [Equation] ̸
[Equation]
[Equation] where [[ · ]] is the Iverson bracket notation as suggested by Knuth [26] and the cost c can be furtherextended to an instance-dependent function c ( x ) .
Our goal is to train a classifier that can minimizethe expectation of ℓ 01 c over the data distribution:
[Equation]
Let us denote by f ∗ = argmin
f R 01 c ( f ) the Bayes optimal solution and η ( x )
= { p ( y | x ) } K y =1 theposterior probabilities.
When evaluated by ℓ 01 c , a classifier receives a standard classification error in{ 0 , 1 } if it makes a prediction and a cost of c if it does not make a prediction (i.e., chooses the rejectoption).
Intuitively, an optimal solution f ∗ should balance the possibility of misclassification and therejection cost c .
This explanation is theoretically justified by Chow’s rule [10]:
Definition 1.
(Chow’s Rule)
A classifier f : X → Y ® is the optimal solution of (1) if and only if it meets the following condition almost surely:
[Equation]
Chow’s rule shows that the optimal solution should refrain from making a decision if the mostcompetent prediction of an example is still not confident enough given a rejection cost c .
2.2 Calibrated Surrogate Losses
Most classification problems can be formalized as the minimization of the target risk, which is theexpectation of a target loss.
Then empirical risk minimization (ERM) is conducted to obtain modelswith performance guarantees.
However, most of the target losses are discontinuous, e.g., the zero-oneloss in multi-class classification and the Hamming/ranking loss in multi-label classification [19].
Therefore, directly optimizing them is usually difficult and even NP-hard [17].
In order to optimize the target risk efficiently, surrogate risk minimization is preferred that minimizingthe expectation of a continuous surrogate loss instead, e.g., the hinge loss in binary classificationand the cross entropy loss in multi-class classification.
For the statistical consistency of learning,calibration [46] is considered as a basic requirement for surrogate losses, which is a pointwise versionof consistency and means that the minimization of the surrogate loss yields that of the target loss foreach possible sample.
A commonly adopted definition of the calibration of surrogates in multi-classclassification is given as follows:
Definition 2.
( ℓ 01 -Calibration [7, 47, 41])
For a K -class classification problem with target loss ℓ 01 , we say Φ : R K × Y → R + is ℓ 01 -calibrated if for any p ∈ ∆ K :
[Equation]
where Φ ( u ) = { Φ( u , y )
[Equation]
The definition of ℓ 01 -calibration requires that a surrogate loss should be able to distinguish betweenoptimal solutions and non-optimal ones w.r.t.
any potential posterior distribution p .
This property [Table] is shown to be a necessary and sufficient condition for the statistical consistency of surrogate riskminimization, and fruitful research on the verification of ℓ 01 -calibrated surrogates has been conducted[7, 56, 47, 41, 40, 18].
Besides multi-class classification, the calibration of surrogate losses also has been studied in variousaspects of statistical learning, including but not limited to, multi-label classification [19, 55, 27, 51],
AUC optimization [20, 34], general linear-fractional utility maximization [3], cost-sensitive learning[11, 44], top- K classification [29, 53], and adversarially robust classification [4, 2, 1].
2.3 Consistency in Classification with Rejection
In the field of CwR, we are also interested in the consistency of surrogate losses.
Let C ⊂ R d whered ∈ N and Φ :
C × Y → R + is a surrogate loss, the consistency is defined as follows:
Definition 3.
( ℓ 01 c -Consistency)
A surrogate loss Φ : C × Y → R + is ℓ 01 c -consistent if thereexists a function φ : C → Y ® for all probability distributions and all the sequences of functions{ g i } i ∈ N : X → C :
[Equation]
where R ϕ ( g ) =
E p ( x ,y )
[Φ( g ( x ) , y )] , R ∗ Φ = inf g : X→C R Φ ( g ) , and R ∗ 01 c = inf f : X→Y ® R 01 c ( f ) .
This definition is inspired by the problem of general multi-class classification [41].
For an ℓ 01 cconsistent surrogate loss Φ , we can safely minimize the surrogate risk R Φ instead while remainingthe consistency guarantee of R 01 c .
To ensure the consistency of Φ , it is routine to discuss the calibration of surrogate losses.
However,unlike the classical multi-class classification problem, where φ is usually an argmax operator, thedesign of φ in the field of CwR can be quite complicated and hard to be unified, which makes itdifficult to directly conduct calibration analysis on Φ .
The flexibility of φ also limits the discussionsto specific types of surrogate losses.
In Ramaswamy et al.
[42], the authors considered the multi-classextensions of the hinge-loss with a confidence threshold.
Ni et al.
[38] indicated that the confidencebased method is indispensable and only focuses on class probability estimation via surrogate riskminimization.
Both of Mozannar and Sontag [35] and Charoenphakdee et al.
[8] gave surrogatelosses for the zero-one- c loss that does not depend on the accurate estimation of the class probability,while Mozannar and Sontag [35] focused on a variant of the cross entropy loss and Charoenphakdeeet al.
[8] constructed calibrated surrogate losses with the ensemble of K calibrated losses for binaryclassification.
In this paper, instead of directly discussing the calibration of surrogate Φ , we show that there is anequivalence between classical multi-class classification and CwR. Based on this equivalence, weshow that it is sufficient for Φ to be ℓ 01 c -consistent by letting it be a simple variant of any calibratedsurrogate loss w.r.t.
the traditional zero-one loss ℓ 01 .
The comparison of the proposed method andrelated works is shown in Table 1.
Equivalence between Classification with Rejection and OrdinaryClassification
In this section, we first show that the risk R 01 c ( f ) can be formalized as a ( K +1) -class classificationproblem, and show that we can obtain ℓ 01 c -consistent surrogates with a variant of any calibratedsurrogate w.r.t.
ℓ 01 , which enables the use of C ⊂ R K +1 and φ ( · )= argmax ( · ) as in the traditionalmulti-class classification tasks.
We also show that such equivalence also holds when the cost cdepends on sample x .
The proof of the conclusions in this section can be found in Appendix A. We start by considering the following distribution D ® c over X × Y ® with probability density ˜
p
( x , ˜ y ) :
Definition 4.
(Self-Augmented Distribution)
A distribution D ® c is called a c -self-augmented distribution w.r.t.
D if its probability density meets the following conditions:
[Equation]
It can be seen that distribution D ® c shares the same marginal density of x as the original distribution
D while D ® c has an augmented class ® with class possibility determined by the rejection cost c .
Based on the connection between D ® c and D , we can further explore the relation between the twotasks: classification on D ® c and CwR on D .
Theorem 1.
For any classifier f : X → Y ® , the following equation holds:
[Equation]
where ˜ R 01 ( f ) =
E ˜ p
( x , ˜ y )
[ ℓ 01 ( f ( x ) , ˜ y )] and ˜ R ∗ 01 = inf f : X→Y ® ˜
R 01 ( f ) .
This equation reveals the equivalence between the two tasks in a straightforward manner.
Sincethe multiplication of the classification risk on D ® c with a positive constant is equal to R 01 c ( f ) , theminimization of ˜ R 01 ( f ) immediately yields the minimization of R 01 c ( f ) and vice versa.
Furthermore,according to the linear correlation between ˜ R 01 ( f ) and R 01 c ( f ) , we can directly quantify the excesserror R 01 c ( f ) − R ∗ 01 c by bounding ˜ R 01 ( f ) − ˜ R c 01 , which is an easier work thanks to the existingresearch of multi-class classification.
In conclusion, risk minimization with ˜ R 01 ( f ) can also give aclassifier with a rejection option with the optimality guarantee, and then we can consider a surrogaterisk minimization problem for multi-class classification instead of CwR.
When the cost c ( x ) is an instance-dependent function, we show that such equivalence still holds with aminor modification.
Considering the reweighted zero-one loss: ¯ ℓ 01 ( f ( x ) , y )
=
(2 − c ( x ))
[[ f ( x ) ̸ = y ]]and its expectation ¯ R 01 ( f ) on D ® c , we have the following conclusion:
Corollary 1.
For any classifier f : X → Y ® , the following inequalities holds:
[Equation]
It is obvious that Lemma 1 is a special case of Lemma 1 with constant cost functions.
Though herewe consider a reweighted classification task, the calibration result of multi-class surrogate losses canstill be applied without any modification since the minimization of ¯ R 01 ( f ) can be seen as ordinaryclassification risk minimization with a slightly different marginal density p
′ ( x ) , which does not affectthe calibration result since the class-posterior possibilities remain unchanged.
All the conclusions inthe rest of this paper can be extended to the scenario of instance-dependent cost and we provide themin Appendix G.
ℓ 01 c -Consistent Surrogates with Arbitrary ℓ 01 -Calibrated Losses
According to the discussions in Section 3, CwR can be safely replaced by multi-class classificationon a special distribution D ® c .
Following the practice of surrogate risk minimization in multi-classclassification, we can replace the zero-one loss ℓ 01 with a surrogate risk Φ :
[-+
 × Y ∪ {-]R K +1 ×Y ∪{ K +1 } → R + and minimizing the surrogate risk with a score-based classifier g : X → R K +1 instead, which isdefined as follows:
[Equation]
where t (˜ y ) =
K + 1 if ˜ y
=
® and t (˜ y )
=
˜
y otherwise.
(3) is a typical formulation of the multiclass classification risk and we can asymptotically minimize it following the ERM framework [48].
After the risk minimization process, the prediction is generated with the following link function φ :[-R K +
 → Y ® :-]
[Equation]
With a properly chosen surrogate Φ , the minimization of ˜ R Φ ( g ) can lead to that of ˜ R 01 ( φ ( g )) , whichindicates the minimization of R 01 c ( φ g ) according to Lemmas 1 and 1.
The theory of how to findsuch surrogates has been thoroughly studied in the field of the classification-calibration of multi-classsurrogates [56, 47, 41].
However, we do not have direct access toward D ® c though it is closely related to the available datadistribution D .
In this section, we propose a family of surrogate losses based on the conclusionsin the previous section, which allows the use of any multi-class classification surrogates.
With thisformulation of surrogates, we can recover the classification risk of ˜ R Φ ( g ) without access to D ® c bytaking its expectation in D .
Based on the loss formulation, we also provide the estimation error boundto show the validity of ERM.
4.1 Formulation of Surrogates
Here, we begin with the definition of a family of surrogates for the zero-one- c loss, and then show howit can relate D and D ® c .
With any multi-class classification loss Φ , we have the following formulationof surrogate losses for ℓ 01 c :
Definition 5.
Given a pre-defined rejection cost c , we have the following formulation of surrogate
L Φ c : R K
+1 × Y → R + for CwR:
[Equation]
where Φ : R K +1 × Y ∪ { K + 1 } → R + and u ∈ R K
+1 .
The proposed surrogate loss is the linear combination of a ( K +1) dimensional multi-class classification loss with coefficient determined by the predefined cost c .
It can also be learned from Appendix
A.2 of Charoenphakdee et al.
[8] that when Φ is the softmax cross entropy loss, (4) is equivalent to
Mozannar and Sontag [35].
The following theorem reveals the connection between ˜ R Φ ( g ) and theexpectation of L Φ c on D :
Theorem 2.
For any g : X → R K +1 and R L Φ c ( g ) =
E p ( x ,y )
[ L Φ c ( g ( x ) , y )] :
[Equation]
The proof is provided in Appendix B. From Theorem 2, we can obtain the risk ˜ R
Φ ( g ) without accessto D ® c with the use of the proposed surrogate (4).
Following the common practice, we can finallyconduct ERM [48] that minimizes the unbiased estimator of R L Φ c ( g ) , which is also that of ˜ R Φ ( g )according to Theorem 2:
[Equation]
After minimizing ˆ R L Φ c ( g ) and obtaining the empirically optimal ˆ g , we can use it for predicting withthe link function φ ◦ ˆ g , where φ ◦ ˆ g ( x ) =
φ (ˆ g ( x )) .
According to the unbiasedness of (3), it is promising that the induced prediction rule φ ◦ ˆ g canapproximate Chow’s rule (Definition 1).
To quantify such approximation, there remains two questions:what is the relation between the minimization of empirical risk ˆ R L Φ c ( g ) and R L Φ c ( g ) , and whetherthe minimization of R L Φ c ( g ) yields that of R 01 c ( φ ◦ g ) .
We will answer the two problems in Section4.2 and Section 5, respectively.
4.2 Estimation Error Bound
In Section 4.1, we proposed a family of surrogates that can recover the surrogate risk on D ® c withonly D and provided an ERM framework to learn the empirically optimal ˆ g .
Here we further justifythe use of ERM by showing that the minimization of ˆ R L Φ c can also result in that of R L Φ c with thefollowing estimation error bound.
Theorem 3.
For any δ ∈ (0 , 1) , suppose the model class of g y is G y and g ∈ G , where G y ⊂ X
→ Rand G ⊂ X → R K +1 is composed of {G y } K +1 y =1 .
Φ( · , y ) is ρ -Lipschitz continuous and is boundedby C Φ > 0 .
Assume that the identifiable condition holds, i.e., min g ∈G R L Φ c ( g )
=
R ∗ L Φ c , then thefollowing inequality holds with probability at least 1 − δ :
[Equation]
where R n ( G y ) is the Rademacher complexity [5] w.r.t.
G y on the distribution with density p ( x ) thatoften decays in the rate of O ( 1 n ) .
We prove this conclusion in Appendix C. From the theorem above, we can learn that with theidentifiable condition which is a common assumption with the use of complex models [4, 25, 31],
R L Φ c (ˆ g ) converges to R ∗ L Φ c in O p (1 / √ n ) , which is the optimal parametric convergence rate withoutadditional assumptions [33].
According to Theorem 2, it is straightforward that ˜ R Φ ( g ) P
→
˜
R ∗ Φ alsoholds.
Nevertheless, the relation between the minimization of surrogate risk ˜ R Φ ( g ) and that of thetarget risk ˜ R 01 ( φ ◦ g ) is still unknown.
According to Lemma 1, the minimization of ˜ R 01 ( φ ◦ g ) isequivalent to zero-one- c risk minimization, which is the goal of CwR. We answer this question in thenext section by giving a necessary and sufficient condition for the ℓ 01 c -consistency for L Φ c .
Theoretical Analysis
In this section, we first point out the necessary and sufficient condition for L Φ c to be ℓ 01 c -calibrated.
Then we further specify the regret transfer bounds for a family of CPE-free surrogates [57], whichhas not been provided with theoretical analysis before.
5.1 Necessary and Sufficient Condition for ℓ 01 c -Consistency
Given the loss formulation (4), a natural idea is to construct surrogate L Φ c with commonly usedmulti-class loss functions.
However, the ℓ 01 c -consistency of such surrogates still remains unchecked.
Here, we show that we can borrow the calibration analyses of multi-class surrogates and set Φ to any( K +1) -class
ℓ 01 -calibrated surrogates according to the following necessary and sufficient condition:
Theorem 4.
L Φ c is ℓ 01 c -consistent for any c ∈ [0 , 1] if and only if Φ is an ℓ 01 -calibrated surrogateloss.
The complete proof is shown in Appendix D and here we provide its sketch.
The equivalence between
CwR and multi-class classification on D c shown in Lemmas 1, 1, and Theorem 2 directly yields thesufficiency of this condition.
Though the equivalent classification problem is limited on D c , ˜ p ( y | x )can be any valid class-posterior probabilities due to the arbitrariness of c and thus the calibration of Φis necessary.
As a result, we can use any Φ in an off-the-shelf manner, i.e., to the consistency of different L Φ c , weonly have to check if Φ is ℓ 01 -calibrated, which has been studied thoroughly [7, 47, 41], instead of tedious case-based discussions.
Furthermore, there is also no need for the consideration of any otherpotential Φ since ℓ 01 -calibration is also necessary.
5.2 Calibration Result for Generalized Cross Entropy Loss
Given the necessary and sufficient condition for ℓ 01 c -consistency, we can construct L Φ c with anyℓ 01 -calibrated surrogates.
However, it has been shown in Charoenphakdee et al.
[8] that it can leadto a model that rejects more data than necessary if the cross entropy (CE) loss is used as Φ , whichis a popular choice as a surrogate.
Another common surrogate is the mean absolute error (MAE).
Though it can avoid CPE and only focus on the crucial class with the maximum posterior possibility,it usually takes more training epochs before convergence [57], which can be costly in practical use.
Here, we consider the generalized cross entropy (GCE) loss [57] that can take the advantages of the
CE loss and MAE, which is defined as below:
Definition 6.
(Generalized cross entropy losses)
For any γ ∈ (0 , 1] , the GCE loss is defined as below:
[Equation]
where S ( · ) is the softmax-transformation.
It can be seen that the loss formulation is equivalent to MAE if γ = 1 and it is also reported in Zhangand Sabuncu [57] that the GCE loss can approximate the CE loss if γ → 0 .
Though the GCE loss hasproved to be effective in practical use, to the best of our knowledge, its calibration results remainunknown, and thus it is unsafe directly combining it with L Φ c .
Theorem 5.
The GCE loss Φ γ is ℓ 01 -calibrated for any γ ∈ (0 , 1] .
For the optimal model
[Equation]
for all the x ∈ X almost surely if γ ∈ (0 , 1) .
If γ = 1 ,
S ( [Equation] = 1 .
The proof can be found in Appendix E.
After verifying the calibration result of the GCE loss,we can combine it with the loss formulation L Φ c and obtain an ℓ 01 c -consistent surrogate.
We willexperimentally demonstrate its effectiveness in the next section.
Experiments
In this section, we provide the experiment results of CwR with deep models, which are evaluated bythe zero-one- c loss following the common practice [38, 8].
We also show the misclassification rateof the accepted data and the ratio of the rejected data.
Details of the setup and the experiments forinstance-dependent cost can be found in Appendix F and G, respectively.
Datasets and Models.
In the experiments, we evaluate the proposed methods and baselines on threewidely-used benchmarks Fashion-MNIST [52], SVHN [37], CIFAR-10 [28] with cost c selected from{ 0 . 05 , 0 . 06 , 0 . 07 , 0 .
10 } for Fashion-MNIST and { 0 . 05 , 0 .
25 , 0 . 30 } theother two.
We conduct data augmentation for CIFAR-10 and use the original datasets of FashionMNIST and SVHN in the experiments.
For Fashion-MNIST, we use a CNN defined in Charoenphakdee et al.
[8], and ResNet-18 and ResNet-34 [24] are used for SVHN and CIFAR-10, respectively.
Baselines.
We compare our method with state-of-the-art methods in CwR, including confidencebased cross entropy loss (CE) [38], learning to defer (DEFER) [35], and cost-sensitive learning-basedmethod with sigmoid loss (CS) [8], in which DEFER is a special case of our method that use crossentropy loss as Φ .
For CE, we also conduct the temperature scaling [23] to alleviate overconfidence .
For the proposed method, we use GCE with default parameter γ = 0 .
7 as suggested in Zhang and Sabuncu [57] and pairwise-sigmoid (Sigmoid) loss [56] to construct the surrogate L Φ c .
We implemented all the methods by Pytorch [39], and conducted all the experiments on NVIDIA
GeForce 3090 GPUs.
[Table]
Experimental Results.
As can be seen from the experimental results reported in Table 2, ourproposed method (i.e., either GCE or Sigmoid) significantly outperforms other compared methodsin most cases.
Obviously, for all the datasets and cost c , our GCE method outperforms the baseline
DEFER method, which indicates that CwR cannot be simply solved by the methods used for learningto defer.
It can be also seen that confidence-based CE is only comparable to the proposed methodon FMNIST with a simple CNN.
When complex models are used, the effect of overconfidence isinevitable even with the use of temperature scaling, which can be induced from the fact that CE oftenrejects less data than GCE on SVHN and CIFAR-10.
Though CS is comparable to GCE on CIFAR-10when the rejection cost is high, its performance degrades drastically when the classification costdecreases, which shows that it is not the best choice in highly error-critical tasks.
When ResNet-18and ResNet-34 are used on SVHN and CIFAR-10 respectively, our GCE method outperforms oris comparable to all the baselines, which shows that GCE is more stable on complex models.
Ourproposed Sigmoid method performs better than most baselines and is comparable to CE with theuse of a simple CNN model, which aligns with the existing observations that pairwise losses areoften effective with simple models [49, 14].
These results show that our method can benefit from theflexibility of the choices of loss functions.
Conclusion
In this paper, we studied the problem of classification with rejection, which can refrain from makinga prediction to avoid critical misclassification.
We derived a novel formulation for CwR that canbe equipped with arbitrary loss functions while maintaining the theoretical guarantees, makingthem highly adaptive to the dataset in practical use.
First, we showed the equivalence between
K -class CwR and a ( K +1) -class
classification problem, and proposed an empirical risk minimizationformulation to solve this problem with an estimation error bound.
Then, we pointed out a necessaryand sufficient condition for the learning consistency of the surrogates constructed on our proposedformulation equipped with any classification-calibrated multi-class losses.
Finally, experimentalresults demonstrated the effectiveness of our proposed method.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix H.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix H.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See the
supplemental material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 6 and Appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Table 2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 6.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 6.
(b) Did you mention the license of the assets?
[N/A]
The used datasets are open bench-
marks.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Please refer to the supplemental materials.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]