Generalizing Consistent Multi-Class Classification with Rejection to be Compatible with Arbitrary Losses
Abstract
Classification with rejection (CwR) refrains from making a prediction to avoid critical misclassification when encountering test samples that are difficult to classify.
Though previous methods for CwR have been provided with theoretical guarantees, they are only compatible with certain loss functions, making them not flexible enough when the loss needs to be changed with the dataset in practice.
In this paper, we derive a novel formulation for CwR that can be equipped with arbitrary loss functions while maintaining the theoretical guarantees.
First, we show that K -class CwR is equivalent to a ( K +1) -class classification problem on the original data distribution with an augmented class, and propose an empirical risk minimization formulation to solve this problem with an estimation error bound.
Then, we find necessary and sufficient conditions for the learning consistency of the surrogates constructed on our proposed formulation equipped with any classification-calibrated multi-class losses, where consistency means the surrogate risk minimization implies the target risk minimization for CwR.
Finally, experimental results validate the effectiveness of our proposed method.
Introduction
In risk-sensitive multi-class classification applications (e g., medical diagnosis, healthcare, autonomous driving and product inspections [13, 22, 44]), misclassification can cause serious or even fatal consequences.
To alleviate this issue, many studies have been conducted on classification with rejection (CwR) [11, 6, 63, 13, 14, 16, 22, 52, 48, 44, 8] which can abstain from making an unsure prediction to prevent such critical misclassification.
Most of the previous studies follow the framework that provides the reject option with a pre-defined cost c which is lower than the misclassification cost 1 .
Given cost c , the problem is further formulated as a risk minimization problem that aims to minimize the expectation of the zero-one- c loss, i.e., the zero-one- c risk.
With the risk minimization process, the obtained classifier can balance the cost of rejection and prediction by choosing to incur a rejection cost c if the misclassification risk is high.
Due to the discontinuous nature of the zero-one- c loss, recent works focused on finding its continuous surrogates to make the optimization problem tractable.
A basic requirement for surrogate losses is the consistency [65, 7, 54, 47], i.e., the surrogate risk minimization implies the zero-one- c risk minimization.
Moreover, compared with the traditional K -class classification task where decisions
[Figure]
are normally made from the index of the maximum coordinate of a K -dimensional scoring function, the design of decision criteria in the CwR task is more elusive due to the existence of a reject option.
By adopting different classification and rejection criteria, various surrogates of the zero-one- c loss have been proposed with consistency analyses [6, 63, 13, 14, 48, 44, 8].
Classical studies focused on developing confidence-based methods [6, 63, 48, 44], which use the outputs of classifiers as confidence values and set a real-valued threshold as the rejection rule.
Representative methods [63, 44] used surrogates that depend on class-posterior probability estimation (CPE) [50, 58], which is challenging when using deep models [25].
Though some of them [6, 48, 35, 24] could avoid CPE, most of them applied the modification of non-differentiable hinge/ramp-like surrogates, and their performance was only validated with linear models.
To avoid the use of the confidence threshold, Cortes et al.
[13] provided an upper bound of the zeroone- c loss as the surrogate that allows the use of a separated rejector and can be trained simultaneously with the classifier, which is regarded as classifier-rejector methods .
Though these methods achieved state-of-the-art performance in binary classification scenarios, they only provided a consistency guarantee for hinge-like and exponential losses and cannot be directly generalized to the multi-class scenario as shown in Ni et al.
Charoenphakdee et al.
[8] showed that K -class CwR can be decomposed into K binary cost-sensitive classification problems [17, 51, 12] and proposed a family of surrogates are the ensembles of arbitrary binary classification losses, which can avoid CPE and the use of confidence threshold with properly chosen losses when the cost function is constant.
Mozannar and Sontag [41] provided a modified version of the cross entropy loss as the surrogate for the task of learning to defer [41, 42] that can also be used in CwR, while its optimal solution still relies on CPE.
In summary, previous works only took limited types of losses into consideration, and there lacks a theoretically grounded framework that can cover all the surrogates used in multi-class classification.
In this paper, we propose a novel framework for CwR that allows the use of arbitrary surrogate losses used in traditional multi-class classification as long as they are classification-calibrated, including but not limited to the well-known cross entropy loss, mean absolute error, focal loss [33, 9], and the pairwise/one-versus-all generalizations of binary margin losses [65].
Thanks to the flexible choices of losses, we are free of the restricted analyses on the consistency of certain surrogates.
The access to a full range of surrogates also make it possible to custom the loss contingent on the characteristics and actual requirements of different datasets and tasks.
An overview of our framework is shown in
Figure 1.
We summarize the main contributions of this work as follows:
• We disclose the equivalence between K -class CwR and a ( K +1) -class classification problem on the original data distribution with an augmented class, by showing the equality between their classification risks.
• We propose a formulation of surrogates for ℓ 01 c that can recover the surrogate risk of a ( K +1) -class classification task only with the K -class training distribution, and derive an estimation error bound for its empirical risk minimization.
• We find a sufficient condition for the consistency of the proposed family of surrogates w.r.t.
the zero-one- c loss that allows the use of any calibrated multi-class surrogates and show its necessity when considering a large family of loss functions.
• We for the first time provide an analysis on the calibration of the generalized cross entropy loss [66] that benefits from both the cross entropy loss and mean absolute error, and experimentally demonstrate that it is suitable for our proposed framework
Preliminaries
In this section, we provide preliminary knowledge of CwR and calibrated surrogate losses, and discuss the consistency in CwR.
2.1 Classification with Rejection
The problem setting of CwR is based on the cost-based framework [11].
Let us denote by X the feature space, Y = { 1 , 2 , . .
, K } the label space , and Y ® = { 1 , 2 , . . .
, K, ® } the label space with a reject option.
We are given instance-label pairs { ( x i , y i ) } ni =1 independently and identically drawn from an underlying distribution with probability density p ( x , y ) .
The goal of CwR is to train a classifier f : X → Y ® that can abstain from making a decision, where ® denotes the reject option.
The evaluation metric of this task is the zero-one- c loss ℓ 01 c , which can be expressed as a variant of the traditional zero-one loss ℓ 01 ( f ( x ) , y ) =
[[ f ( x ) ̸ = y ]] :
[Equation]
where [[ · ]] is the Iverson bracket notation as suggested by Knuth [29] and the cost c can be further extended to an instance-dependent function c ( x ) .
Our goal is to train a classifier that can minimize the expectation of ℓ 01 c over the data distribution:
[Equation]
Let us denote by f ∗ = argmin
f R 01 c ( f ) the Bayes optimal solution and η ( x ) = { p ( y | x ) } K y {+=
+}
the posterior probabilities.
When evaluated by ℓ 01 c , a classifier receives a standard classification error in { 0 , 1 } if it makes a prediction and a cost of c if it does not make a prediction (i.e., chooses the reject option).
Intuitively, an optimal solution f ∗ should balance the possibility of misclassification and the rejection cost c .
This explanation is theoretically justified by Chow’s rule [11]:
Definition 1.
(Chow’s Rule)
A classifier f : X → Y ® is the optimal solution of (1) if and only if it meets the following condition almost surely:
[Equation]
Chow’s rule shows that the optimal solution should refrain from making a decision if the most confident prediction of an example is still not confident enough given a rejection cost c .
2.2 Calibrated Surrogate Losses
Most classification problems can be formalized as the minimization of the target risk, which is the expectation of a target loss.
Then empirical risk minimization (ERM) is conducted to obtain models with performance guarantees.
However, most of the target losses are discontinuous, e.g., the zero-one loss in multi-class classification and the Hamming/ranking loss in multi-label classification [20].
Therefore, directly optimizing them is usually difficult and even NP-hard [18].
In order to optimize the target risk efficiently, surrogate risk minimization is preferred that minimizing the expectation of a continuous surrogate loss instead, e.g., the hinge loss in binary classification and the cross entropy loss in multi-class classification.
For the statistical consistency of learning, calibration [53] is considered as a basic requirement for surrogate losses, which is a pointwise version of consistency and means that the minimization of the surrogate loss yields that of the target loss for each possible sample.
A commonly adopted definition of the calibration of surrogates in multi-class classification is given as follows:
Definition 2.
( ℓ 01 -Calibration [7, 54, 47])
For a K -class classification problem with target loss ℓ 01 , we say Φ : R K × Y → R + is ℓ 01 -calibrated if for any p ∈ ∆ K :
[Equation]
where Φ ( u ) = { Φ( u , y ) } Ky =1 , L 01 ( u ) = { ℓ 01 ( argmax y [-0 
 Y-]′ ∈Y u y ′ , y ) }
Ky
=1 .
The definition of ℓ 01 -calibration requires that a surrogate loss should be able to distinguish between optimal solutions and non-optimal ones w.r.t.
any potential posterior distribution p .
This property is shown to be a necessary and sufficient condition for the statistical consistency of surrogate risk minimization, and fruitful research on the verification of ℓ 01 -calibrated surrogates has been conducted [7, 65, 54, 47, 46, 19].
[Table]
Besides multi-class classification, the calibration of surrogate losses also has been studied in various aspects of statistical learning, including but not limited to, multi-label classification [20, 64, 30, 59],
AUC optimization [21, 39], general linear-fractional utility maximization [3], cost-sensitive learning [12, 51], top- K classification [32, 62], and adversarially robust classification [4, 2, 1].
2.3 Consistency in Classification with Rejection
‘ In the field of CwR, we are also interested in the consistency of surrogate losses.
Let C ⊂ R d where d ∈ N and Φ : C × Y
→ R + is a surrogate loss, the consistency is defined as follows:
Definition 3.
( ℓ 01 c -Consistency)
A surrogate loss Φ : C × Y → R + is ℓ 01 c
-consistent if there exists a function φ : C → Y ® for all probability distributions and all the sequences of functions { g i } i ∈ N : X → C :
[Equation]
[Equation]
where R ϕ ( g ) =
E p ( x ,y )
[Φ( g ( x ) , y )] , R ∗ Φ = inf g : X→C R Φ ( g ) , and R ∗ 01 c = inf f : X→Y ® R 01 c ( f ) .
This definition is inspired by the problem of general multi-class classification [47].
For an ℓ 01 cconsistent surrogate loss Φ , we can safely minimize the surrogate risk R Φ instead while retaining the consistency guarantee of R 01 c .
To ensure the consistency of Φ , it is routine to discuss the calibration of surrogate losses.
However, unlike the classical multi-class classification problem, where φ is usually an argmax operator, the design of φ for CwR can be quite complicated and hard to be unified, which makes it difficult to directly do calibration analysis on Φ .
The flexibility of φ also limits the discussions to specific types of surrogate losses.
In Ramaswamy et al.
[48], the authors considered the multi-class extensions of the hinge-loss with a confidence threshold.
Ni et al.
[44] indicated that the confidence-based method is indispensable and only focuses on class probability estimation via surrogate risk minimization.
Both of Mozannar and Sontag [41] and Charoenphakdee et al.
[8] gave surrogate losses for the zero-one- c loss that does not depend on the accurate estimation of the class probability, while Mozannar and
Sontag [41] focused on a variant of the cross entropy loss and Charoenphakdee et al.
[8] constructed calibrated surrogate losses with the ensemble of K calibrated losses for binary classification.
In this paper, instead of directly discussing the calibration of surrogate Φ , we show that there is an equivalence between classical multi-class classification and CwR. Based on this equivalence, we show that it is sufficient for Φ to be ℓ 01 c -consistent by letting it be a simple variant of any calibrated surrogate loss w.r.t.
the traditional zero-one loss ℓ 01 .
The comparison of the proposed method and related works is shown in Table 1.
Equivalence between Classification with Rejection and Ordinary Classification
In this section, we first show that the risk R 01 c ( f ) can be formalized as a ( K +1) -class classification problem, and show that we can obtain ℓ 01 c -consistent surrogates with a variant of any calibrated surrogate w.r.t.
ℓ 01 , which enables the use of C ⊂ R K +1 and φ ( · )= argmax ( · ) as in the traditional multi-class classification tasks.
We also show that such equivalence also holds when the cost c depends on sample x .
The proof of the conclusions in this section can be found in Appendix A.
We start by considering the following distribution D ® c over X × Y ® with probability density ˜
p
( x , ˜ y ) :
Definition 4.
(Self-Augmented Distribution)
A distribution D ® c is called a c -self-augmented distribution w.r.t.
D if its probability density meets the following conditions:
[Equation]
It can be seen that distribution D ® c shares the same marginal density of x as the original distribution
D while D ® c has an augmented class ® with class probability determined by the rejection cost c .
Based on the connection between D ® c and D , we can further explore the relation between the two tasks: classification on D ® c and CwR on D .
Theorem 1.
For any classifier f : X → Y ® , the following equation holds:
[Equation]
where ˜ R 01 ( f ) =
E ˜ p
( x , ˜ y )
[ ℓ 01 ( f ( x ) , ˜ y )] and ˜ R [-⇤ 
-]∗ 01 = inf f : X→Y ® ˜
R 01 ( f ) .
Theorem 1 reveals the equivalence between the two tasks in a straightforward manner.
Since the multiplication of the classification risk on D ® c with a positive constant is equal to R 01 c ( f ) , the minimization of ˜ R 01 ( f ) immediately yields the minimization of R 01 c ( f ) and vice versa.
Furthermore, according to the linear correlation between ˜ R 01 ( f ) and R 01 c ( f ) , we can directly quantify the excess error R 01 c ( f ) − R ∗ 01 c by bounding ˜ R 01 ( f ) − ˜ R c 01 , which is an easier work thanks to the existing research of multi-class classification.
In conclusion, risk minimization with ˜ R 01 ( f ) can also give a classifier with a rejection option with the optimality guarantee, and then we can consider a surrogate risk minimization problem for multi-class classification instead of CwR.
When the cost c ( x ) is an instance-dependent function, we show that such equivalence still holds with a minor modification.
Considering the reweighted zero-one loss: ¯ ℓ 01 ( f ( x ) , y ) =
(2 − c ( x ))
[[ f ( x ) ̸ = y ]] and its expectation ¯ R 01 ( f ) on D ® c , we have the following conclusion:
Corollary 1.
For any classifier f : X → Y ® , the following equation holds:
[Equation]
It is obvious that Theorem 1 is a special case of Corollary 1 with constant cost functions.
Though here we consider a reweighted classification task, the calibration result of multi-class surrogate losses can still be applied without any modification since the minimization of ¯ R 01 ( f ) can be seen as ordinary classification risk minimization with a slightly different marginal density p ′ ( x ) , which does not affect the calibration result since the class-posterior possibilities remain unchanged.
All the conclusions in the rest of this paper can be extended to the scenario of instance-dependent cost (see Appendix G).
ℓ 01 c -Consistent Surrogates with Arbitrary ℓ 01 -Calibrated Losses
As discussed in Section 3, CwR can be safely replaced by multi-class classification on a special distribution D ® c .
Following the surrogate risk minimization in multi-class classification, we can replace the zero-one loss ℓ 01 with a surrogate risk Φ : R K {++
+}
× Y ∪ { K + 1 } → R + and minimizing the surrogate risk with a score-based classifier g : X → R K +1 instead, which is defined as follows:
[Equation]
where t (˜ y ) =
K + 1 if ˜ y
=
® and t (˜ y )
=
˜
y otherwise.
(3) is a typical formulation of the multiclass classification risk and we can asymptotically minimize it following the ERM framework [55].
After the risk minimization process, the prediction is generated with the following link function φ : R K {++
+}
→ Y ®
[Equation]
With a classification-calibrated surrogate Φ , it is sufficient to say that the minimization of ˜ R Φ ( g ) can lead to that of ˜ R 01 ( φ ( g )) , which indicates the minimization of R 01 c ( φ ( g )) according to Theoremand Corollary 1.
The theory of how to find such surrogates has been thoroughly studied in the field of the classification-calibration of multi-class surrogates [65, 54, 47].
However, we do not have direct access toward D ® c though it is closely related to the available data distribution D .
In this section, we propose a family of surrogate losses based on the conclusions in the previous section, which allows the use of any multi-class classification surrogates.
With this formulation of surrogates, we can recover the classification risk of ˜ R Φ ( g ) without access to D ® c by taking its expectation in D .
Based on the loss formulation, we also provide the estimation error bound to show the validity of ERM.
4.1 Formulation of Surrogates
Here, we begin with the definition of a family of surrogates for the zero-one- c loss, and then show how it can relate D and D ® c .
With any multi-class classification loss Φ , we have the following formulation of surrogate losses for ℓ 01 c :
Definition 5.
Given a pre-defined rejection cost c , we have the following formulation of surrogate
L Φ c : R K
+1 × Y → R + for CwR:
[Equation]
(4) where Φ : R K +1 × Y ∪ { K + 1 } → R + and u ∈ R K +1 .
The proposed surrogate loss is the linear combination of a ( K +1) dimensional multi-class classification loss with coefficient determined by the predefined cost c .
It can also be learned from Appendix
A.2 of Charoenphakdee et al.
[8] that when Φ is the softmax cross entropy loss, (4) is equivalent to
Mozannar and Sontag [41].
The following theorem reveals the connection between ˜ R Φ ( g ) and the expectation of L Φ c on D :
Theorem 2.
For any g : X → R K +1 and R L Φ c ( g ) =
E p ( x ,y )
[ L Φ c ( g ( x ) , y )] :
[Equation]
The proof is provided in Appendix B. From Theorem 2, we can obtain the risk ˜ R Φ ( g ) without access to D ® c with the use of the proposed surrogate (4).
Following the common practice, we can finally conduct ERM [55] that minimizes the unbiased estimator of R L Φ c ( g ) , which is also that of ˜ R Φ ( g ) according to Theorem 2:
[Equation]
After minimizing ˆ R L Φ c ( g ) and obtaining the empirically optimal ˆ g , we can use it for predicting with the link function φ ◦ ˆ g , where φ ◦ ˆ g ( x ) = φ (ˆ g ( x )) .
According to the unbiasedness of (3), it is promising that the induced prediction rule φ ◦ ˆ g can approximate Chow’s rule (Definition 1).
To quantify such approximation, there remains two questions: what is the relation between the minimization of empirical risk ˆ R L Φ c ( g ) and R L Φ c ( g ) , and whether the minimization of R L Φ c ( g ) yields that of R 01 c ( φ ◦ g ) .
We will answer the two problems in Section 4.2 and Section 5, respectively.
4.2 Estimation Error Bound
In Section 4.1, we proposed a family of surrogates that can recover the surrogate risk on D ® c with only
D and provided an ERM framework to learn the empirically optimal ˆ g = min g ∈G ˆ R L Φ c ( g ) .
Here we further justify the use of ERM by showing that the minimization of ˆ R L Φ c can also result in that of
R L Φ c with the following estimation error bound.
Theorem 3.
For any δ ∈ (0 , 1) , suppose the model class of g y is G y and g ∈ G , where G y ⊂ X
→ R and G ⊂ X → R K +1 is composed of {G y } K +1 y =1 .
Φ( · , y ) is ρ -Lipschitz continuous and is bounded by C Φ > 0 .
Assume that the identifiable condition holds, i.e., min g ∈G R L Φ c ( g ) =
R ∗ L Φ c , then the following inequality holds with probability at least 1 − δ :
[Equation]
where R n ( G y ) is the Rademacher complexity [5] w.r.t.
G y on the distribution with density p ( x ) that often decays in the rate of O ( 1 √ n ) .
We prove this conclusion in Appendix C. From the theorem above, we can learn that with the identifiable condition which is a common assumption with the use of complex models [4, 27, 34],
R L Φ c (ˆ g ) converges to R ∗ L Φ c in O p (1 / √ n ) , which is the optimal parametric convergence rate without additional assumptions [38].
According to Theorem 2, it is straightforward that ˜ R Φ ( g ) P →
˜
R ∗ Φ also holds.
Nevertheless, the relation between the minimization of surrogate risk ˜ R Φ ( g ) and that of the target risk ˜ R 01 ( φ ◦ g ) is still unknown.
According to Theorem 1, the minimization of ˜ R 01 ( φ ◦ g ) is equivalent to zero-one- c risk minimization, which is the goal of CwR.
We answer this question in the next section by giving a sufficient condition for the ℓ 01 c -consistency for L Φ c .
Theoretical Analysis
In this section, we first point out the sufficient condition for L Φ c to be ℓ 01 c -calibrated and show that it is also necessary if the candidates of Φ are permutation-invariant, which consists of a large number of commonly used loss functions, including but not limited to cross-entropy loss, focal loss, and mean absolute error.
Then we further specify the regret transfer bounds for a family of CPE-free surrogates [66], which has not been provided with theoretical analysis before.
5.1 Necessary and Sufficient Conditions for ℓ 01 c -Consistency
Given the loss formulation (4), a natural idea is to construct surrogate L Φ c with commonly used multi-class loss functions.
Here, we justify this idea by showing that we can borrow the calibration analyses of multi-class surrogates and set Φ to any ( K +1) -class
ℓ 01 -calibrated surrogates according to the following conditions:
Theorem 4.
L Φ c is ℓ 01 c -consistent for any c ∈ [0 , 1] if Φ is an ℓ 01 -calibrated surrogate loss.
Furthermore, if Φ is permutation-invariant, i.e., Φ( P g ) =
P Φ( g ) for all permutation matrices P , Φ ’s ℓ 01 -calibration is also necessary for the ℓ 01 c -consistency of L Φ c .
The complete proof is shown in Appendix D and here we provide its sketch.
The equivalence between
CwR and multi-class classification on D ® c shown in Theorem 1, 2, and Corollary 1 directly yields the sufficiency of this condition.
Though the equivalent classification problem is limited on D ® c , the permutation-invariance of Φ and the arbitrariness of c require that the minimizers of the expectation of Φ should be included in that of ℓ 01 for any potential distributions, which indicates the necessity of Φ ’s calibration.
As a result, we can use any Φ in an off-the-shelf manner, i.e., to the consistency of different L Φ c , we only have to check if Φ is ℓ 01 -calibrated, which has been studied thoroughly [7, 54, 47], instead of tedious case-based discussions.
It is noticeable that when considering multi-class surrogates that are not permutation-invariant, the calibration of Φ may not be necessary, which indicates that surrogates that are not calibrated and not permutation-invariant may also be potential options.
Nevertheless, a classification-calibrated surrogate Φ can always be a safe choice given the sufficiency in Theoremand fruitful calibration analyses.
5.2 Calibration Result for Generalized Cross Entropy Loss
Given the sufficient condition for ℓ 01 c -consistency, we can construct L Φ c with any ℓ 01 -calibrated surrogates.
However, it has been shown in Charoenphakdee et al.
[8] that it can lead to a model that rejects more data than necessary if the cross entropy (CE) loss is used as Φ , which is a popular choice as a surrogate.
Another common surrogate is the mean absolute error (MAE).
Though it can avoid
CPE and only focus on the crucial class with the maximum posterior probability, it usually takes more training epochs before convergence [66], which can be costly in practical use.
Here, we consider the generalized cross entropy (GCE) loss [66] that can take the advantages of the
CE loss and MAE, which is defined as below:
Definition 6.
(Generalized cross entropy losses)
For any γ ∈ (0 , 1] , the GCE loss is defined as below:
[Equation]
where S ( · ) is the softmax-transformation.
It can be seen that the loss formulation is equivalent to MAE if γ = [-
-]1 and it is also reported in Zhang and Sabuncu [66] that the GCE loss can approximate the CE loss if γ → 0 .
Though the GCE loss has proved to be effective in practical use, to the best of our knowledge, its calibration results remain unknown.
To justify the combination of GCE loss and L Φ c , we give the calibration analysis and further show its analytical solution.
Theorem 5.
The GCE loss Φ γ is ℓ 01 -calibrated for any γ ∈ (0 , 1] .
For the optimal model g ∗ , S ( g ∗ ) y
=
η 11 − γ y /
[Equation]
for all the x ∈ X almost surely if γ ∈ (0 , 1) .
If γ = 1 ,
S ( g ∗ ) argmax y η y = 1 .
The proof can be found in Appendix E.
After verifying the calibration result of the GCE loss, we can combine it with the loss formulation L Φ c and obtain an ℓ 01 c -consistent surrogate.
We will experimentally demonstrate its effectiveness in the next section.
Experiments
In this section, we provide the experiment results of CwR with deep models, which are evaluated by the zero-one- c loss following the common practice [44, 8].
We also show the misclassification rate of the accepted data and the ratio of the rejected data.
Details of the setup and the experiments for instance-dependent cost can be found in Appendix F and G, respectively.
Datasets and Models.
In the experiments, we evaluate the proposed methods and baselines on three widely-used benchmarks Fashion-MNIST [60], SVHN [43], CIFAR-10 [31] with cost c selected from { 0 . 05 , 0 . 06 , 0 . 07 , 0 .
10 } for Fashion-MNIST and { 0 . 05 , 0 .
30 } the other two.
We conduct data augmentation for CIFAR-10 and use the original datasets of FashionMNIST and SVHN in the experiments.
For Fashion-MNIST, we use a CNN defined in Charoenphakdee et al.
[8], and ResNet-18 and ResNet-34 [26] are used for SVHN and CIFAR-10, respectively.
Baselines.
We compare our method with state-of-the-art methods in CwR, including confidencebased cross entropy loss (CE) [44], learning to defer (DEFER) [41], and cost-sensitive learning-based method with sigmoid loss (CS) [8], in which DEFER is a special case of our method that use cross entropy loss as Φ .
For CE, we also conduct the temperature scaling [25] to alleviate overconfidence.
For the proposed method, we use GCE with default parameter γ = 0 .
7 as suggested in Zhang and Sabuncu [66] and pairwise-sigmoid (Sigmoid) loss [65] to construct the surrogate L Φ c .
We implemented all the methods by Pytorch [45], and conducted all the experiments on NVIDIA
GeForce 3090 GPUs.
Experimental Results.
As can be seen from the experimental results reported in Table 2, our proposed method (i.e., either GCE or Sigmoid) significantly outperforms other compared methods in most cases.
Obviously, for all the datasets and cost c , our GCE method outperforms the baseline
DEFER method, which indicates that CwR cannot be simply solved by the methods used for learning to defer.
It can be also seen that confidence-based CE is only comparable to the proposed method on FMNIST with a simple CNN.
When complex models are used, the effect of overconfidence is inevitable even with the use of temperature scaling, which can be induced from the fact that CE often rejects less data than GCE on SVHN and CIFAR-10.
Though CS is comparable to GCE on
CIFAR-10 when the rejection cost is high, its performance degrades drastically when the rejection cost decreases, which shows that it is not the best choice in highly error-critical tasks.
When ResNetand ResNet-34 are used on SVHN and CIFAR-10 respectively, our GCE method outperforms or is comparable to all the baselines, which shows that GCE is more stable on complex models.
Our proposed Sigmoid method performs better than most baselines and is comparable to CE with the use of a simple CNN model, which aligns with the existing observations that pairwise losses are
[Table]
often effective with simple models [57, 15].
These results show that our method can benefit from the flexibility of the choices of loss functions.
Conclusion
In this paper, we studied the problem of classification with rejection, which can refrain from making a prediction to avoid critical misclassification.
We derived a novel formulation for CwR that can be equipped with arbitrary loss functions while maintaining the theoretical guarantees, making them highly adaptive to the dataset in practical use.
First, we showed the equivalence between
K -class CwR and a ( K +1) -class classification problem, and proposed an empirical risk minimization formulation to solve this problem with an estimation error bound.
Then, we pointed out necessary and sufficient conditions for the learning consistency of the surrogates constructed on our proposed formulation equipped with any classification-calibrated multi-class losses.
Finally, experimental results demonstrated the effectiveness of our proposed method.
Acknowledgement
This research is supported by the National Research Foundation, Singapore under its Industry Alignment Fund – Pre-positioning (IAF-PP)
Funding Initiative.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of
National Research Foundation, Singapore.
Yuzhou Cao was also supported by Ant Group through
Ant Research Intern Program.
Lei Feng was supported by the National Natural Science Foundation of China (Grant No. 62106028), Chongqing Overseas Chinese Entrepreneurship and Innovation
Support Program, and CAAI-Huawei MindSpore Open Fund.
MS was supported by JST CREST
Grant Number JPMJCR18A2.