Generalizing Consistent Multi-Class Classiﬁcation with Rejection to be Compatible with Arbitrary Losses
Abstract
Classiﬁcation with rejection (CwR) refrains from making a prediction to avoid critical misclassiﬁcation when encountering test samples that are difﬁcult to classify.
Though previous methods for CwR have been provided with theoretical guarantees, they are only compatible with certain loss functions, making them not ﬂexible enough when the loss needs to be changed with the dataset in practice.
In this paper, we derive a novel formulation for CwR that can be equipped with arbitrary loss functions while maintaining the theoretical guarantees.
First, we show that K -class CwR is equivalent to a ( K +1) -class classiﬁcation problem on the original data distribution with an augmented class, and propose an empirical risk minimization formulation to solve this problem with an estimation error bound.
Then, we ﬁnd necessary and sufﬁcient conditions for the learning consistency of the surrogates constructed on our proposed formulation equipped with any classiﬁcation-calibrated multi-class losses, where consistency means the surrogate risk minimization implies the target risk minimization for CwR.
Finally, experimental results validate the effectiveness of our proposed method.
Introduction
In risk-sensitive multi-class classiﬁcation applications (e.g., medical diagnosis, healthcare, autonomous driving, and product inspections [12, 21, 43]), misclassiﬁcation can cause serious or even fatal consequences.
To alleviate this issue, many studies have been conducted on classiﬁcation with rejection (CwR) [10, 6, 61, 12, 13, 15, 21, 51, 47, 43, 8], which can abstain from making an unsure prediction to prevent such critical misclassiﬁcation.
Most of the previous studies follow the framework that provides the reject option with a pre-deﬁned cost c which is lower than the misclassiﬁcation cost 1 .
Given cost c , the problem is further formulated as a risk minimization problem that aims to minimize the expectation of the zero-one- c loss, i.e., the zero-one- c risk.
With the risk minimization process, the obtained classiﬁer can balance the cost of rejection and prediction by choosing to incur a rejection cost c if the misclassiﬁcation risk is high. Due to the discontinuous nature of the zero-one- c loss, recent works focused on ﬁnding its continuous surrogates to make the optimization problem tractable. A basic requirement for surrogate losses is the consistency [63, 7, 53, 46], i.e., the surrogate risk minimization implies the zero-one- c risk minimization. Moreover, compared with the traditional K -class classiﬁcation task where decisions
[Figure]
are normally made from the index of the maximum coordinate of a K -dimensional scoring function, the design of decision criteria in the CwR task is more elusive due to the existence of a reject option.
By adopting different classiﬁcation and rejection criteria, various surrogates of the zero-one- c loss have been proposed with consistency analyses [6, 61, 12, 13, 47, 43, 8].
Classical studies focused on developing conﬁdence-based methods [6, 61, 47, 43], which use the outputs of classiﬁers as conﬁdence values and set a real-valued threshold as the rejection rule.
Representative methods [61, 43] used surrogates that depend on class-posterior probability estimation (CPE) [49, 56], which is challenging when using deep models [24].
Though some of them [6, 47, 34, 23] could avoid CPE, most of them applied the modiﬁcation of non-differentiable hinge/ramp-like surrogates, and their performance was only validated with linear models.
To avoid the use of the conﬁdence threshold, Cortes et al.
[12] provided an upper bound of the zeroone- c loss as the surrogate that allows the use of a separated rejector and can be trained simultaneously with the classiﬁer, which is regarded as classiﬁer-rejector methods .
Though these methods achieved state-of-the-art performance in binary classiﬁcation scenarios, they only provided a consistency guarantee for hinge-like and exponential losses and cannot be directly generalized to the multi-class scenario as shown in Ni et al.
Charoenphakdee et al.
[8] showed that K -class CwR can be decomposed into K binary cost-sensitive classiﬁcation problems [16, 50, 11] and proposed a family of surrogates are the ensembles of arbitrary binary classiﬁcation losses, which can avoid CPE and the use of conﬁdence threshold with properly chosen losses when the cost function is constant.
Mozannar and Sontag [40] provided a modiﬁed version of the cross entropy loss as the surrogate for the task of learning to defer [40, 41] that can also be used in CwR, while its optimal solution still relies on CPE.
In summary, previous works only took limited types of losses into consideration, and there lacks a theoretically grounded framework that can cover all the surrogates used in multi-class classiﬁcation.
In this paper, we propose a novel framework for CwR that allows the use of arbitrary surrogate losses used in traditional multi-class classiﬁcation as long as they are classiﬁcation-calibrated, including but not limited to the well-known cross entropy loss, mean absolute error, focal loss [32, 9], and the pairwise/one-versus-all generalizations of binary margin losses [63].
Thanks to the ﬂexible choices of losses, we are free of the restricted analyses on the consistency of certain surrogates.
The access to a full range of surrogates also make it possible to custom the loss contingent on the characteristics and actual requirements of different datasets and tasks.
An overview of our framework is shown in
Figure 1.
We summarize the main contributions of this work as follows:
• We disclose the equivalence between K -class CwR and a ( K +1) -class classiﬁcation problem on the original data distribution with an augmented class, by showing the equality between their classiﬁcation risks.
• We propose a formulation of surrogates for ` 01 c that can recover the surrogate risk of a ( K +1) -class classiﬁcation task only with the K -class training distribution, and derive an estimation error bound for its empirical risk minimization.
• We ﬁnd a sufﬁcient condition for the consistency of the proposed family of surrogates w.r.t.
the zero-one- c loss that allows the use of any calibrated multi-class surrogates and show its necessity when considering a large family of loss functions.
• We for the ﬁrst time provide an analysis on the calibration of the generalized cross entropy loss [64] that beneﬁts from both the cross entropy loss and mean absolute error, and experimentally demonstrate that it is suitable for our proposed framework.
Preliminaries
In this section, we provide preliminary knowledge of CwR and calibrated surrogate losses, and discuss the consistency in CwR.
2.1 Classiﬁcation with Rejection
The problem setting of CwR is based on the cost-based framework [10].
Let us denote by X the feature space, Y = { 1 , 2 , . .
, K } the label space , and Y ® = { 1 , 2 , . . .
, K, ® } the label space with a reject option.
We are given instance-label pairs { ( x i , y i ) } ni =1 independently and identically drawn from an underlying distribution with probability density p ( x , y ) .
The goal of CwR is to train a classiﬁer f : X ! Y ® that can abstain from making a decision, where ® denotes the reject option.
The evaluation metric of this task is the zero-one- c loss ` 01 c , which can be expressed as a variant of the traditional zero-one loss ` 01 ( f ( x ) , y ) =
[[ f ( x [Equation] 6 = y ]] :
[Equation]
where [[ · ]] is the Iverson bracket notation as suggested by Knuth [28] and the cost c can be further extended to an instance-dependent function c ( x ) .
Our goal is to train a classiﬁer that can minimize the expectation of ` 01 c over the data distribution:
[Equation]
Let us denote by f ⇤ = argmin
f R 01 c ( f ) the Bayes optimal solution and ⌘ ( x ) = { p ( y | x ) } K y = the posterior probabilities.
When evaluated by ` 01 c , a classiﬁer receives a standard classiﬁcation error in { 0 , 1 } if it makes a prediction and a cost of c if it does not make a prediction (i.e., chooses the reject option).
Intuitively, an optimal solution f ⇤ should balance the possibility of misclassiﬁcation and the rejection cost c .
This explanation is theoretically justiﬁed by Chow’s rule [10]:
Deﬁnition 1.
(Chow’s Rule)
A classiﬁer f : X ! Y ® is the optimal solution of (1) if and only if it meets the following condition almost surely:
[Equation]
Chow’s rule shows that the optimal solution should refrain from making a decision if the most conﬁdent prediction of an example is still not conﬁdent enough given a rejection cost c .
2.2 Calibrated Surrogate Losses
Most classiﬁcation problems can be formalized as the minimization of the target risk, which is the expectation of a target loss.
Then empirical risk minimization (ERM) is conducted to obtain models with performance guarantees.
However, most of the target losses are discontinuous, e.g., the zero-one loss in multi-class classiﬁcation and the Hamming/ranking loss in multi-label classiﬁcation [19].
Therefore, directly optimizing them is usually difﬁcult and even NP-hard [17].
In order to optimize the target risk efﬁciently, surrogate risk minimization is preferred that minimizing the expectation of a continuous surrogate loss instead, e.g., the hinge loss in binary classiﬁcation and the cross entropy loss in multi-class classiﬁcation.
For the statistical consistency of learning, calibration [52] is considered as a basic requirement for surrogate losses, which is a pointwise version of consistency and means that the minimization of the surrogate loss yields that of the target loss for each possible sample.
A commonly adopted deﬁnition of the calibration of surrogates in multi-class classiﬁcation is given as follows:
Deﬁnition 2.
( ` 01 -Calibration [7, 53, 46])
For a K -class classiﬁcation problem with target loss ` 01 , we say  : R K ⇥ Y ! R + is ` 01 -calibrated if for any p 2  K :
[Equation]
, where  ( u ) = {  ( u , y ) } Ky =1 , L 01 ( u ) = { ` 01 ( argmax y {+0 
 Y+} u y 0 , y ) }
Ky
=1 .
The deﬁnition of ` 01 -calibration requires that a surrogate loss should be able to distinguish between optimal solutions and non-optimal ones w.r.t.
any potential posterior distribution p .
This property is shown to be a necessary and sufﬁcient condition for the statistical consistency of surrogate risk minimization, and fruitful research on the veriﬁcation of ` 01 -calibrated surrogates has been conducted [7, 63, 53, 46, 45, 18]. [Table]
Besides multi-class classiﬁcation, the calibration of surrogate losses also has been studied in various aspects of statistical learning, including but not limited to, multi-label classiﬁcation [19, 62, 29, 57],
AUC optimization [20, 38], general linear-fractional utility maximization [3], cost-sensitive learning [11, 50], top- K classiﬁcation [31, 60], and adversarially robust classiﬁcation [4, 2, 1].
2.3 Consistency in Classiﬁcation with Rejection
In the ﬁeld of CwR, we are also interested in the consistency of surrogate losses.
Let C ⇢ R d where d 2 N and  : C ⇥ Y !
R + is a surrogate loss, the consistency is deﬁned as follows:
Deﬁnition 3.
( ` 01 c -Consistency)
A surrogate loss  : C ⇥ Y ! R + is ` 01 c -consistent if there exists a function ' : C ! Y ® for all probability distributions and all the sequences of functions { g i } i 2 N : [Equation]
[Equation]
[Equation]
(2) where R  ( g ) =
E p ( x ,y )
[  ( g ( x ) , y )] , R ⇤  = inf g : X ! C R  ( g ) , and
[Equation] This deﬁnition is inspired by the problem of general multi-class classiﬁcation [46].
For an ` 01 cconsistent surrogate loss  , we can safely minimize the surrogate risk R  instead while retaining the consistency guarantee of R 01 c .
To ensure the consistency of  , it is routine to discuss the calibration of surrogate losses.
However, unlike the classical multi-class classiﬁcation problem, where ' is usually an argmax operator, the design of ' for CwR can be quite complicated and hard to be uniﬁed, which makes it difﬁcult to directly do calibration analysis on  .
The ﬂexibility of ' also limits the discussions to speciﬁc types of surrogate losses.
In Ramaswamy et al.
[47], the authors considered the multi-class extensions of the hinge-loss with a conﬁdence threshold.
Ni et al.
[43] indicated that the conﬁdence-based method is indispensable and only focuses on class probability estimation via surrogate risk minimization.
Both of Mozannar and Sontag [40] and Charoenphakdee et al.
[8] gave surrogate losses for the zero-one- c loss that does not depend on the accurate estimation of the class probability, while Mozannar and
Sontag [40] focused on a variant of the cross entropy loss and Charoenphakdee et al.
[8] constructed calibrated surrogate losses with the ensemble of K calibrated losses for binary classiﬁcation.
In this paper, instead of directly discussing the calibration of surrogate  , we show that there is an equivalence between classical multi-class classiﬁcation and CwR. Based on this equivalence, we show that it is sufﬁcient for  to be ` 01 c -consistent by letting it be a simple variant of any calibrated surrogate loss w.r.t.
the traditional zero-one loss ` 01 .
The comparison of the proposed method and related works is shown in Table 1.
Equivalence between Classiﬁcation with Rejection and Ordinary Classiﬁcation
In this section, we ﬁrst show that the risk R 01 c ( f ) can be formalized as a ( K +1) -class classiﬁcation problem, and show that we can obtain ` 01 c -consistent surrogates with a variant of any calibrated surrogate w.r.t.
` 01 , which enables the use of C ⇢ R K +1 and ' ( · )= argmax ( · ) as in the traditional multi-class classiﬁcation tasks.
We also show that such equivalence also holds when the cost c depends on sample x .
The proof of the conclusions in this section can be found in Appendix A. We start by considering the following distribution D ® c over X ⇥ Y ® with probability density ˜
p
( x , ˜ y ) :
Deﬁnition 4.
(Self-Augmented Distribution)
A distribution D ® c is called a c -self-augmented distribution w.r.t.
D if its probability density meets the following conditions:
[Equation]
It can be seen that distribution D ® c shares the same marginal density of x as the original distribution
D while D ® c has an augmented class ® with class probability determined by the rejection cost c .
Based on the connection between D ® c and D , we can further explore the relation between the two tasks: classiﬁcation on D ® c and CwR on D .
Theorem 1.
For any classiﬁer f : X ! Y ® , the following equation holds:
[Equation]
where ˜ R 01 ( f ) =
E ˜ p
( x , ˜ y )
[ ` 01 ( f ( x ) , ˜ y )] and ˜ R {+⇤ 
+} = inf f : X ! Y ® ˜
R 01 ( f ) .
Theorem 1 reveals the equivalence between the two tasks in a straightforward manner.
Since the multiplication of the classiﬁcation risk on D ® c with a positive constant is equal to R 01 c ( f ) , the minimization of ˜ R 01 ( f ) immediately yields the minimization of R 01 c ( f ) and vice versa.
Furthermore, according to the linear correlation between ˜ R 01 ( f ) and R 01 c ( f ) , we can directly quantify the excess error R 01 c ( f )  R ⇤ 01 c by bounding ˜ R 01 ( f )  ˜ R c 01 , which is an easier work thanks to the existing research of multi-class classiﬁcation.
In conclusion, risk minimization with ˜ R 01 ( f ) can also give a classiﬁer with a rejection option with the optimality guarantee, and then we can consider a surrogate risk minimization problem for multi-class classiﬁcation instead of CwR.
When the cost c ( x ) is an instance-dependent function, we show that such equivalence still holds with a minor modiﬁcation.
Considering the reweighted zero-one loss: ¯ ` 01 ( f ( x ) , y ) =
(2  c ( x ))
[[ f ( x ) 6 = y ]] and its expectation ¯ R 01 ( f ) on D ® c , we have the following conclusion:
Corollary 1.
For any classiﬁer f : X ! Y ® , the following equation holds:
[Equation]
It is obvious that Theorem 1 is a special case of Corollary 1 with constant cost functions.
Though here we consider a reweighted classiﬁcation task, the calibration result of multi-class surrogate losses can still be applied without any modiﬁcation since the minimization of ¯ R 01 ( f ) can be seen as ordinary classiﬁcation risk minimization with a slightly different marginal density p 0 ( x ) , which does not affect the calibration result since the class-posterior possibilities remain unchanged.
All the conclusions in the rest of this paper can be extended to the scenario of instance-dependent cost (see Appendix G).
` 01 c -Consistent Surrogates with Arbitrary ` 01 -Calibrated Losses
As discussed in Section 3, CwR can be safely replaced by multi-class classiﬁcation on a special distribution D ® c .
Following the surrogate risk minimization in multi-class classiﬁcation, we can replace the zero-one loss ` 01 with a surrogate risk  : R K +1 ⇥
[-+
-]Y [ { K + 1 } !
R + and minimizing the surrogate risk with a score-based classiﬁer g : [Equation]
instead, which is deﬁned as follows:
[Equation]
where t (˜ y ) =
K + 1 if ˜ y
=
® and t (˜ y )
=
˜
y otherwise.
(3) is a typical formulation of the multiclass classiﬁcation risk and we can asymptotically minimize it following the ERM framework [54].
After the risk minimization process, the prediction is generated with the following link function ' : R K [-+
-]+1 !
Y ® :
[Equation]
[Equation]
, argmax
[Equation]
argmax [Equation] u y ( x ) , else . With a classiﬁcation-calibrated surrogate  , it is sufﬁcient to say that the minimization of ˜ R  ( g ) can lead to that of ˜ R 01 ( ' ( g )) , which indicates the minimization of R 01 c ( ' ( g )) according to Theoremand Corollary 1.
The theory of how to ﬁnd such surrogates has been thoroughly studied in the ﬁeld of the classiﬁcation-calibration of multi-class surrogates [63, 53, 46].
However, we do not have direct access toward D ® c though it is closely related to the available data distribution D .
In this section, we propose a family of surrogate losses based on the conclusions in the previous section, which allows the use of any multi-class classiﬁcation surrogates.
With this formulation of surrogates, we can recover the classiﬁcation risk of ˜ R  ( g ) without access to D ® c by taking its expectation in D .
Based on the loss formulation, we also provide the estimation error bound to show the validity of ERM.
4.1 Formulation of Surrogates
Here, we begin with the deﬁnition of a family of surrogates for the zero-one- c loss, and then show how it can relate D and D ® c .
With any multi-class classiﬁcation loss  , we have the following formulation of surrogate losses for ` 01 c :
Deﬁnition 5.
Given a pre-deﬁned rejection cost c , we have the following formulation of surrogate
L  c : R K +1 ⇥
[Equation]
+ for CwR:
[Equation]
where  : R K +1 ⇥ Y [ { K + 1 } ! R + and u 2 R K +1 .
The proposed surrogate loss is the linear combination of a ( K +1) dimensional multi-class classiﬁcation loss with coefﬁcient determined by the predeﬁned cost c .
It can also be learned from Appendix
A.2 of Charoenphakdee et al.
[8] that when  is the softmax cross entropy loss, (4) is equivalent to
Mozannar and Sontag [40].
The following theorem reveals the connection between ˜ R  ( g ) and the expectation of L  c on D :
Theorem 2.
For any g : X ! R K +1 and R L  c ( g ) =
E p ( x ,y )
[ L  c ( g ( x ) , y )] :
[Equation]
The proof is provided in Appendix B. From Theorem 2, we can obtain the risk ˜ R  ( g ) without access to D ® c with the use of the proposed surrogate (4).
Following the common practice, we can ﬁnally conduct ERM [54] that minimizes the unbiased estimator of R L  c ( g ) , which is also that of ˜ R  ( g ) according to Theorem 2:
[Equation]
After minimizing ˆ R L  c ( g ) and obtaining the empirically optimal ˆ g , we can use it for predicting with the link function '  ˆ g , where '  ˆ g ( x ) = ' (ˆ g ( x )) .
According to the unbiasedness of (3), it is promising that the induced prediction rule '  ˆ g can approximate Chow’s rule (Deﬁnition 1).
To quantify such approximation, there remains two questions: what is the relation between the minimization of empirical risk ˆ R L  c ( g ) and R L  c ( g ) , and whether the minimization of R L  c ( g ) yields that of R 01 c ( '  g ) .
We will answer the two problems in Section 4.2 and Section 5, respectively.
4.2 Estimation Error Bound
In Section 4.1, we proposed a family of surrogates that can recover the surrogate risk on D ® c with only
D and provided an ERM framework to learn the empirically optimal ˆ g = min g 2 G ˆ R L  c ( g ) .
Here we further justify the use of ERM by showing that the minimization of ˆ R L  c can also result in that of
R L  c with the following estimation error bound.
Theorem 3.
For any  2 (0 , 1) , suppose the model class of g y is G y and g 2 G , where G y ⇢ X !
R and G ⇢ X ! R K +1 is composed of {G y } K +1 y =1 .
( · , y ) is ⇢ -Lipschitz continuous and is bounded by C  > 0 .
Assume that the identiﬁable condition holds, i.e., min g 2 G R L  c ( g ) =
R ⇤ L  c , then the following inequality holds with probability at least 1   :
[Equation]
where R n ( G y ) is the Rademacher complexity [5] w.r.t.
G y on the distribution with density p ( x ) that often decays in the rate of O ( 1 p n ) .
We prove this conclusion in Appendix C. From the theorem above, we can learn that with the identiﬁable condition which is a common assumption with the use of complex models [4, 26, 33],
R L  c (ˆ g ) converges to R ⇤ L  c in O p (1 / p n ) , which is the optimal parametric convergence rate without additional assumptions [37].
According to Theorem 2, it is straightforward that ˜ R  ( g ) P !
˜
R ⇤  also holds.
Nevertheless, the relation between the minimization of surrogate risk ˜ R  ( g ) and that of the target risk ˜ R 01 ( '  g ) is still unknown.
According to Theorem 1, the minimization of ˜ R 01 ( '  g ) is equivalent to zero-one- c risk minimization, which is the goal of CwR. We answer this question in the next section by giving a sufﬁcient condition for the ` 01 c -consistency for L  c .
Theoretical Analysis
In this section, we ﬁrst point out the sufﬁcient condition for L  c to be ` 01 c -calibrated and show that it is also necessary if the candidates of  are permutation-invariant, which consists of a large number of commonly used loss functions, including but not limited to cross-entropy loss, focal loss, and mean absolute error.
Then we further specify the regret transfer bounds for a family of CPE-free surrogates [64], which has not been provided with theoretical analysis before.
5.1 Necessary and Sufﬁcient Conditions for ` 01 c -Consistency
Given the loss formulation (4), a natural idea is to construct surrogate L  c with commonly used multi-class loss functions.
Here, we justify this idea by showing that we can borrow the calibration analyses of multi-class surrogates and set  to any ( K +1) -class
` 01 -calibrated surrogates according to the following conditions:
Theorem 4.
L  c is ` 01 c -consistent for any c 2 [0 , 1] if  is an ` 01 -calibrated surrogate loss.
Furthermore, if  is permutation-invariant, i.e.,  ( P g ) =
P  ( g ) for all permutation matrices P ,  ’s ` 01 -calibration is also necessary for the ` 01 c -consistency of L  c .
The complete proof is shown in Appendix D and here we provide its sketch.
The equivalence between
CwR and multi-class classiﬁcation on D ® c shown in Theorem 1, 2, and Corollary 1 directly yields the sufﬁciency of this condition.
Though the equivalent classiﬁcation problem is limited on D ® c , the permutation-invariance of  and the arbitrariness of c require that the minimizers of the expectation of  should be included in that of ` 01 for any potential distributions, which indicates the necessity of  ’s calibration.
As a result, we can use any  in an off-the-shelf manner, i.e., to the consistency of different L  c , we only have to check if  is ` 01 -calibrated, which has been studied thoroughly [7, 53, 46], instead of tedious case-based discussions.
It is noticeable that when considering multi-class surrogates that are not permutation-invariant, the calibration of  may not be necessary, which indicates that surrogates that are not calibrated and not permutation-invariant may also be potential options.
Nevertheless, a classiﬁcation-calibrated surrogate  can always be a safe choice given the sufﬁciency in Theoremand fruitful calibration analyses.
5.2 Calibration Result for Generalized Cross Entropy Loss
Given the sufﬁcient condition for ` 01 c -consistency, we can construct L  c with any ` 01 -calibrated surrogates.
However, it has been shown in Charoenphakdee et al.
[8] that it can lead to a model that rejects more data than necessary if the cross entropy (CE) loss is used as  , which is a popular choice as a surrogate.
Another common surrogate is the mean absolute error (MAE).
Though it can avoid
CPE and only focus on the crucial class with the maximum posterior probability, it usually takes more training epochs before convergence [64], which can be costly in practical use.
Here, we consider the generalized cross entropy (GCE) loss [64] that can take the advantages of the
CE loss and MAE, which is deﬁned as below:
Deﬁnition 6.
(Generalized cross entropy losses)
For any  2 (0 , 1] , the GCE loss is deﬁned as below:
[Equation]
where S ( · ) is the softmax-transformation.
It can be seen that the loss formulation is equivalent to MAE if  = {+
+} and it is also reported in Zhang and Sabuncu [64] that the GCE loss can approximate the CE loss if  ! 0 .
Though the GCE loss has proved to be effective in practical use, to the best of our knowledge, its calibration results remain unknown.
To justify the combination of GCE loss and L  c , we give the calibration analysis and further show its analytical solution.
Theorem 5.
The GCE loss   is ` 01 -calibrated for any  2 (0 , 1] .
For the optimal model g ⇤ , S ( g ⇤ ) y = ⌘ 11   y
[Equation]
for all the x [Equation] almost surely if  2 (0 , 1) .
If  = 1 ,
S ( g ⇤ ) argmax y ⌘ y = 1 .
The proof can be found in Appendix E.
After verifying the calibration result of the GCE loss, we can combine it with the loss formulation L  c and obtain an ` 01 c -consistent surrogate.
We will experimentally demonstrate its effectiveness in the next section.
Experiments
In this section, we provide the experiment results of CwR with deep models, which are evaluated by the zero-one- c loss following the common practice [43, 8].
We also show the misclassiﬁcation rate of the accepted data and the ratio of the rejected data.
Details of the setup and the experiments for instance-dependent cost can be found in Appendix F and G, respectively.
Datasets and Models.
In the experiments, we evaluate the proposed methods and baselines on three widely-used benchmarks Fashion-MNIST [58], SVHN [42], CIFAR -10 [30] with cost c selected from { 0 . 05 , 0 . 06 , 0 . 07 , 0 .
10 } for Fashion-MNIST and { 0 . 05 , 0 .
30 } the other two.
We conduct data augmentation for CIFAR-10 and use the original datasets of FashionMNIST and SVHN in the experiments.
For Fashion-MNIST, we use a CNN deﬁned in Charoenphakdee et al.
[8], and ResNet-18 and ResNet-34 [25] are used for SVHN and CIFAR-10, respectively.
Baselines.
We compare our method with state-of-the-art methods in CwR, including conﬁdencebased cross entropy loss (CE) [43], learning to defer (DEFER) [40], and cost-sensitive learning-based method with sigmoid loss (CS) [8], in which DEFER is a special case of our method that use cross entropy loss as  .
For CE, we also conduct the temperature scaling [24] to alleviate overconﬁdence.
For the proposed method, we use GCE with default parameter  = 0 .
7 as suggested in Zhang and Sabuncu [64] and pairwise-sigmoid (Sigmoid) loss [63] to construct the surrogate L  c .
We implemented all the methods by Pytorch [44], and conducted all the experiments on NVIDIA
GeForce 3090 GPUs.
Experimental Results.
As can be seen from the experimental results reported in Table 2, our proposed method (i.e., either GCE or Sigmoid) signiﬁcantly outperforms other compared methods in most cases.
Obviously, for all the datasets and cost c , our GCE method outperforms the baseline
DEFER method, which indicates that CwR cannot be simply solved by the methods used for learning to defer.
It can be also seen that conﬁdence-based CE is only comparable to the proposed method on FMNIST with a simple CNN.
When complex models are used, the effect of overconﬁdence is inevitable even with the use of temperature scaling, which can be induced from the fact that CE often rejects less data than GCE on SVHN and CIFAR-10.
Though CS is comparable to GCE on
CIFAR-10 when the rejection cost is high, its performance degrades drastically when the rejection cost decreases, which shows that it is not the best choice in highly error-critical tasks.
When ResNetand ResNet-34 are used on SVHN and CIFAR-10 respectively, our GCE method outperforms or is comparable to all the baselines, which shows that GCE is more stable on complex models.
Our proposed Sigmoid method performs better than most baselines and is comparable to CE with the use of a simple CNN model, which aligns with the existing observations that pairwise losses are [Table] often effective with simple models [55, 14].
These results show that our method can beneﬁt from the ﬂexibility of the choices of loss functions.
Conclusion
In this paper, we studied the problem of classiﬁcation with rejection, which can refrain from making a prediction to avoid critical misclassiﬁcation.
We derived a novel formulation for CwR that can be equipped with arbitrary loss functions while maintaining the theoretical guarantees, making them highly adaptive to the dataset in practical use.
First, we showed the equivalence between
K -class CwR and a ( K +1) -class classiﬁcation problem, and proposed an empirical risk minimization formulation to solve this problem with an estimation error bound.
Then, we pointed out necessary and sufﬁcient conditions for the learning consistency of the surrogates constructed on our proposed formulation equipped with any classiﬁcation-calibrated multi-class losses.
Finally, experimental results demonstrated the effectiveness of our proposed method.
Acknowledgement
This research is supported by the National Research Foundation, Singapore under its Industry Alignment Fund – Pre-positioning (IAF-PP)
Funding Initiative.
Any opinions, ﬁndings and conclusions or recommendations expressed in this material are those of the author(s) and do not reﬂect the views of
National Research Foundation, Singapore.
Yuzhou Cao was also supported by Ant Group through
Ant Research Intern Program.
Lei Feng was supported by the National Natural Science Foundation of China (Grant No. 62106028), Chongqing Overseas Chinese Entrepreneurship and Innovation
Support Program, and CAAI-Huawei MindSpore Open Fund.
MS was supported by JST CREST
Grant Number JPMJCR18A2.