Generalizing Consistent Multi-Class Classification with Rejection to be Compatible with Arbitrary Losses
Anonymous Author(s)
AffiliationAddress email
Abstract
Classification with rejection (CwR) refrains from making a prediction to avoidcritical misclassification when encountering test samples that are difficult to classify.
Though previous methods for CwR have been provided with theoreticalguarantees, they are only compatible with certain loss functions, making them notflexible enough when the loss needs to be changed with the dataset in practice.
In this paper, we derive a novel formulation for CwR that can be equipped witharbitrary loss functions while maintaining the theoretical guarantees.
First, weshow that K -class CwR is equivalent to a ( K +1) -class classification problemon the original data distribution with an augmented class, and propose an empirical risk minimization formulation to solve this problem with an estimationerror bound.
Then, we find necessary and sufficient conditions for the learningconsistency of the surrogates constructed on our proposed formulation equippedwith any classification-calibrated multi-class losses, where consistency means thesurrogate risk minimization implies the target risk minimization for CwR. Finally,experimental results validate the effectiveness of our proposed method.
Introduction
In risk-sensitive multi-class classification applications (e g., medical diagnosis, healthcare, autonomous driving, and product inspections [12, 21, 38]), misclassification can cause serious oreven fatal consequences.
To alleviate this issue, many studies have been conducted on classificationwith rejection (CwR) [10, 6, 54, 12, 13, 15, 21, 45, 42, 38, 8] which can abstain from making anunsure prediction to prevent such critical misclassification.
Most of the previous studies follow the framework that provides the reject option with a pre-definedcost c which is lower than the misclassification cost 1 .
Given cost c , the problem is further formulatedas a risk minimization problem that aims to minimize the expectation of the zero-one- c loss, i e., thezero-one- c risk.
With the risk minimization process, the obtained classifier can balance the cost of the nature of the zero-one- c loss recent works focused on finding its continuous surrogates to make the optimization tractable basic requirement for surrogate the i e the surrogate risk minimization implies the zero-one- c risk with the traditional K -class classification task decisions are normally made the index of the maximum coordinate of a K -dimensional scoring function of criteria the task is more elusive due to the existence of a reject
[Figure]
By adopting different classification and rejection criteria, various surrogates of the zero-one- c losshave been proposed with consistency analyses [6, 54, 12, 13, 42, 38, 8].
Classical studies focused on developing confidence-based methods [6, 54, 42, 38], which use theoutputs of classifiers as confidence values and set a real-valued threshold as the rejection rule.
Representative methods [54, 38] used surrogates that depend on class-posterior probability estimation(CPE) [43, 50], which is challenging when using deep models [23].
Though some of them [6, 42, 32,22] could avoid CPE, most of them applied the modification of non-differentiable hinge/ramp-likesurrogates, and their performance was only validated with linear models.
To avoid the use of the confidence threshold, Cortes et al.
[12] provided an upper bound of the zeroone- c loss as the surrogate that allows the use of a separated rejector and can be trained simultaneouslywith the classifier, which is regarded as classifier-rejector methods .
Though these methods achievedstate-of-the-art performance in binary classification scenarios, they only provided a consistencyguarantee for hinge-like and exponential losses and cannot be directly generalized to the multi-classscenario as shown in Ni et al.
Charoenphakdee et al.
[8] showed that K -class CwR can bedecomposed into K binary cost-sensitive classification problems [16, 44, 11] and proposed a familyof surrogates are the ensembles of arbitrary binary classification losses, which can avoid CPE and theuse of confidence threshold with properly chosen losses when the cost function is constant.
Mozannarand Sontag [35] provided a modified version of the cross entropy loss as the surrogate for the task oflearning to defer [35, 36] that can also be used in CwR, while its optimal solution still relies on CPE.
In summary, previous works only took limited types of losses into consideration, and there lacks atheoretically grounded framework that can cover all the surrogates used in multi-class classification.
In this paper, we propose a novel framework for CwR that allows the use of arbitrary surrogate lossesused in traditional multi-class classification as long as they are classification-calibrated, includingbut not limited to the well-known cross entropy loss, mean absolute error, focal loss [30, 9], and thepairwise/one-versus-all generalizations of binary margin losses [56].
Thanks to the flexible choicesof losses, we be free of the restricted analyses on the consistency of certain surrogates.
The access toa full range of surrogates also make it possible to custom the loss contingent on the characteristicsand actual requirements of different datasets and tasks.
An overview of our framework is shown in Figure 1.
We summarize the main contributions of this work as follows:
• We disclose the equivalence between K -class CwR and a ( K +1) -class
classification problemon the original data distribution with an augmented class, by showing the equality between theirclassification risks.
• We propose a formulation of surrogates for ℓ 01 c that can recover the surrogate risk of a ( K +1)
-class classification task only with the K -class training distribution, and derive an estimation errorbound for its empirical risk minimization.
• We find a sufficient condition for the consistency of the proposed family of surrogates w.r.t.
thezero-one- c loss that allows the use of any calibrated multi-class surrogates and show its necessity
when considering a large family of loss functions.
We for the first time provide an analysis on the calibration of the generalized cross entropy loss[57] that benefits from both the cross entropy loss and mean absolute error, and experimentallydemonstrate that it is suitable for our proposed framework.
Preliminaries
In this section, we provide preliminary knowledge of CwR and calibrated surrogate losses, anddiscuss the consistency in CwR.
2.1 Classification with Rejection
The problem setting of CwR is based on the cost-based framework [10].
Let us denote by X the feature space, Y = { 1 , 2 , . .
, K } the label space , and Y ® = { 1 , 2 , . . .
, K, ® } the label space with a reject option.
We are given instance-label pairs { ( x i , y i ) } ni =1 independently and identically drawn from an underlying distribution with probability density p ( x , y ) .
The goal of CwR is to train a classifier f : X → Y ® that can abstain from making a decision, where ® denotes the reject option.
The evaluation metric of this task is the zero-one- c loss ℓ 01 c , which can be expressed as a variant of the traditional zero-one loss ℓ 01 ( f ( x ) , y ) =
[[ f ( x ) ̸ = y ]] :
[Equation]
where [[ · ]] is the Iverson bracket notation as suggested by Knuth [26] and the cost c can be furtherextended to an instance-dependent function c ( x ) .
Our goal is to train a classifier that can minimizethe expectation of ℓ 01 c over the data distribution:
[Equation]
Let us denote by f ∗ = argmin
f R 01 c ( f ) the Bayes optimal solution and η ( x )
= { p ( y | x ) } Ky =1 theposterior probabilities.
When evaluated by ℓ 01 c , a classifier receives a standard classification error in{ 0 , 1 } if it makes a prediction and a cost of c if it does not make a prediction (i.e., chooses the rejectoption).
Intuitively, an optimal solution f ∗ should balance the possibility of misclassification and therejection cost c .
This explanation is theoretically justified by Chow’s rule [10]:
Definition 1.
(Chow’s Rule)
A classifier f : X → Y ® is the optimal solution of (1) if and only if it meets the following condition almost surely:
[Equation]
Chow’s rule shows that the optimal solution should refrain from making a decision if the mostconfident prediction of an example is still not confident enough given a rejection cost c .
2.2 Calibrated Surrogate Losses
Most classification problems can be formalized as the minimization of the target risk, which is theexpectation of a target loss.
Then empirical risk minimization (ERM) is conducted to obtain modelswith performance guarantees.
However, most of the target losses are discontinuous, e.g., the zero-oneloss in multi-class classification and the Hamming/ranking loss in multi-label classification [19].
Therefore, directly optimizing them is usually difficult and even NP-hard [17].
In order to optimize the target risk efficiently, surrogate risk minimization is preferred that minimizingthe expectation of a continuous surrogate loss instead, e.g., the hinge loss in binary classificationand the cross entropy loss in multi-class classification.
For the statistical consistency of learning,calibration [46] is considered as a basic requirement for surrogate losses, which is a pointwise versionof consistency and means that the minimization of the surrogate loss yields that of the target loss foreach possible sample.
A commonly adopted definition of the calibration of surrogates in multi-classclassification is given as follows:
Definition 2.
( ℓ 01 -Calibration [7, 47, 41])
For a K -class classification problem with target loss ℓ 01 , we say Φ : R K × Y → R + is ℓ 01 -calibrated if for any p ∈ ∆ K :
[Equation]
, where Φ ( u ) = { Φ( u , y ) }
Ky =1 , L 01 ( u ) = { ℓ 01 ( argmax y ′ ∈Y u y ′ , y ) }
Ky
=1 . [Table]
The definition of ℓ 01 -calibration requires that a surrogate loss should be able to distinguish betweenoptimal solutions and non-optimal ones w.r.t.
any potential posterior distribution p .
This propertyis shown to be a necessary and sufficient condition for the statistical consistency of surrogate riskminimization, and fruitful research on the verification of ℓ 01 -calibrated surrogates has been conducted[7, 56, 47, 41, 40, 18].
Besides multi-class classification, the calibration of surrogate losses also has been studied in variousaspects of statistical learning, including but not limited to, multi-label classification [19, 55, 27, 51],
AUC optimization [20, 34], general linear-fractional utility maximization [3], cost-sensitive learning[11, 44], top- K classification [29, 53], and adversarially robust classification [4, 2, 1].
2.3 Consistency in Classification with Rejection
In the field of CwR, we are also interested in the consistency of surrogate losses.
Let C ⊂ R d whered ∈ N and Φ :
C × Y → R + is a surrogate loss, the consistency is defined as follows:
Definition 3.
( ℓ 01 c -Consistency)
A surrogate loss Φ : C × Y → R + is ℓ 01 c -consistent if thereexists a function φ : C → Y ® for all probability distributions and all the sequences of functions{ g i } i ∈ N : X → C :
[Equation]
where R ϕ ( g ) =
E p ( x ,y )
[Φ( g ( x ) , y )] , R ∗ Φ = inf g : X→C R Φ ( g ) , and R ∗ 01 c = inf f : X→Y ® R 01 c ( f ) .
This definition is inspired by the problem of general multi-class classification [41].
For an ℓ 01 cconsistent surrogate loss Φ , we can safely minimize the surrogate risk R Φ instead while retaining theconsistency guarantee of R 01 c .
To ensure the consistency of Φ , it is routine to discuss the calibration of surrogate losses.
However,unlike the classical multi-class classification problem, where φ is usually an argmax operator, thedesign of φ for CwR can be quite complicated and hard to be unified, which makes it difficult todirectly do calibration analysis on Φ .
The flexibility of φ also limits the discussions to specific typesof surrogate losses.
In Ramaswamy et al.
[42], the authors considered the multi-class extensions of thehinge-loss with a confidence threshold.
Ni et al.
[38] indicated that the confidence-based method isindispensable and only focuses on class probability estimation via surrogate risk minimization.
Bothof Mozannar and Sontag [35] and Charoenphakdee et al.
[8] gave surrogate losses for the zero-one- closs that does not depend on the accurate estimation of the class probability, while Mozannar and Sontag [35] focused on a variant of the cross entropy loss and Charoenphakdee et al.
[8] constructedcalibrated surrogate losses with the ensemble of K calibrated losses for binary classification.
In this paper, instead of directly discussing the calibration of surrogate Φ , we show that there is anequivalence between classical multi-class classification and CwR. Based on this equivalence, weshow that it is sufficient for Φ to be ℓ 01 c -consistent by letting it be a simple variant of any calibratedsurrogate loss w.r.t.
the traditional zero-one loss ℓ 01 .
The comparison of the proposed method andrelated works is shown in Table 1.
Equivalence between Classification with Rejection and Ordinary
Classification
In this section, we first show that the risk R 01 c ( f ) can be formalized as a ( K +1) -class classificationproblem, and show that we can obtain ℓ 01 c -consistent surrogates with a variant of any calibratedsurrogate w.r.t.
ℓ 01 , which enables the use of C ⊂ R K +1 and φ ( · )= argmax ( · ) as in the traditionalmulti-class classification tasks.
We also show that such equivalence also holds when the cost cdepends on sample x .
The proof of the conclusions in this section can be found in Appendix A. Westart by considering the following distribution D ® c over X × Y ® with probability density ˜
p
( x , ˜ y ) :
Definition 4.
(Self-Augmented Distribution)
A distribution D ® c is called a c -self-augmented distribution w.r.t.
D if its probability density meets the following conditions:
[Equation]
It can be seen that distribution D ® c shares the same marginal density of x as the original distribution
D while D ® c has an augmented class ® with class probability determined by the rejection cost c .
Based on the connection between D ® c and D , we can further explore the relation between the twotasks: classification on D ® c and CwR on D .
Theorem 1.
For any classifier f : X → Y ® , the following equation holds:
[Equation]
where ˜ R 01 ( f ) =
E ˜ p
( x , ˜ y )
[ ℓ 01 ( f ( x ) , ˜ y )] and ˜ R ∗ 01 = inf f : X→Y ® ˜
R 01 ( f ) .
Theorem 1 reveals the equivalence between the two tasks in a straightforward manner.
Sincethe multiplication of the classification risk on D ® c with a positive constant is equal to R 01 c ( f ) , theminimization of ˜ R 01 ( f ) immediately yields the minimization of R 01 c ( f ) and vice versa.
Furthermore,according to the linear correlation between ˜ R 01 ( f ) and R 01 c ( f ) , we can directly quantify the excesserror R 01 c ( f ) − R ∗ 01 c by bounding ˜ R 01 ( f ) − ˜ R c 01 , which is an easier work thanks to the existingresearch of multi-class classification.
In conclusion, risk minimization with ˜ R 01 ( f ) can also give aclassifier with a rejection option with the optimality guarantee, and then we can consider a surrogaterisk minimization problem for multi-class classification instead of CwR.
When the cost c ( x ) is an instance-dependent function, we show that such equivalence still holds with aminor modification.
Considering the reweighted zero-one loss: ¯ ℓ 01 ( f ( x ) , y )
=
(2 − c ( x ))
[[ f ( x ) ̸ = y ]]and its expectation ¯ R 01 ( f ) on D ® c , we have the following conclusion:
Corollary 1.
For any classifier f : X → Y ® , the following equation holds:
[Equation]
It is obvious that Theorem 1 is a special case of Corollary 1 with constant cost functions.
Though herewe consider a reweighted classification task, the calibration result of multi-class surrogate losses canstill be applied without any modification since the minimization of ¯ R 01 ( f ) can be seen as ordinaryclassification risk minimization with a slightly different marginal density p
′ ( x ) , which does not affectthe calibration result since the class-posterior possibilities remain unchanged.
All the conclusions inthe rest of this paper can be extended to the scenario of instance-dependent cost (see Appendix G).
ℓ 01 c -Consistent Surrogates with Arbitrary ℓ 01 -Calibrated Losses
As discussed in Section 3, CwR can be safely replaced by multi-class classification on a specialdistribution D ® c .
Following the surrogate risk minimization in multi-class classification, we canreplace the zero-one loss ℓ 01 with a surrogate risk Φ : R K
{++
 × Y ∪ } K {++ 1 } → R + and minimizingthe surrogate risk with a score-based classifier g : X → R K +1 instead, which is defined as follows:
[Equation]
where t (˜ y ) =
K + 1 if ˜ y
=
® and t (˜ y )
=
˜
y otherwise.
(3) is a typical formulation of the multiclass classification risk and we can asymptotically minimize it following the ERM framework [48].
After the risk minimization process, the prediction is generated with the following link function φ : {+R K +
 → Y ® :+}
[Equation]
With a classification-calibrated surrogate Φ , it is sufficient to say that the minimization of ˜ R Φ ( g ) canlead to that of ˜ R 01 ( φ ( g )) , which indicates the minimization of R 01 c ( φ ( g )) according to Theorem 1and Corollary 1.
The theory of how to find such surrogates has been thoroughly studied in the field ofthe classification-calibration of multi-class surrogates [56, 47, 41].
However, we do not have direct access toward D ® c though it is closely related to the available datadistribution D .
In this section, we propose a family of surrogate losses based on the conclusionsin the previous section, which allows the use of any multi-class classification surrogates.
With thisformulation of surrogates, we can recover the classification risk of ˜ R Φ ( g ) without access to D ® c bytaking its expectation in D .
Based on the loss formulation, we also provide the estimation error boundto show the validity of ERM.
4.1 Formulation of Surrogates
Here, we begin with the definition of a family of surrogates for the zero-one- c loss, and then show howit can relate D and D ® c .
With any multi-class classification loss Φ , we have the following formulationof surrogate losses for ℓ 01 c :
Definition 5.
Given a pre-defined rejection cost c , we have the following formulation of surrogate
L Φ c : R K
+1 × Y → R + for CwR:
[Equation]
[Equation]
where Φ : R K +1 × Y ∪ { K + 1 } → R + and u ∈ R K
+1 .
The proposed surrogate loss is the linear combination of a ( K +1) dimensional multi-class classification loss with coefficient determined by the predefined cost c .
It can also be learned from Appendix
A.2 of Charoenphakdee et al.
[8] that when Φ is the softmax cross entropy loss, (4) is equivalent to
Mozannar and Sontag [35].
The following theorem reveals the connection between ˜ R Φ ( g ) and theexpectation of L Φ c on D :
Theorem 2.
For any g : X → R K +1 and R L Φ c ( g ) =
E p ( x ,y )
[ L Φ c ( g ( x ) , y )] :
[Equation]
The proof is provided in Appendix B. From Theorem 2, we can obtain the risk ˜ R
Φ ( g ) without accessto D ® c with the use of the proposed surrogate (4).
Following the common practice, we can finallyconduct ERM [48] that minimizes the unbiased estimator of R L Φ c ( g ) , which is also that of ˜ R Φ ( g )according to Theorem 2:
[Equation]
After minimizing ˆ R L Φ c ( g ) and obtaining the empirically optimal ˆ g , we can use it for predicting withthe link function φ ◦ ˆ g , where φ ◦ ˆ g ( x ) =
φ (ˆ g ( x )) .
According to the unbiasedness of (3), it is promising that the induced prediction rule φ ◦ ˆ g canapproximate Chow’s rule (Definition 1).
To quantify such approximation, there remains two questions:what is the relation between the minimization of empirical risk ˆ R L Φ c ( g ) and R L Φ c ( g ) , and whetherthe minimization of R L Φ c ( g ) yields that of R 01 c ( φ ◦ g ) .
We will answer the two problems in Section4.2 and Section 5, respectively.
4.2 Estimation Error Bound
In Section 4.1, we proposed a family of surrogates that can recover the surrogate risk on D ® c with only D and provided an ERM framework to learn the empirically optimal ˆ g = min
g ∈G ˆ R L Φ c ( g ) . Here wefurther justify the use of ERM by showing that the minimization of ˆ R L Φ c can also result in that of R L Φ c with the following estimation error bound.
Theorem 3.
For any δ ∈ (0 , 1) , suppose the model class of g y is G y and g ∈ G , where G y ⊂ X
→ Rand G ⊂ X → R K +1 is composed of {G y } K +1 y =1 .
Φ( · , y ) is ρ -Lipschitz continuous and is boundedby C Φ > 0 .
Assume that the identifiable condition holds, i.e., min g ∈G R L Φ c ( g )
=
R ∗ L Φ c , then thefollowing inequality holds with probability at least 1 − δ :
[Equation]
where R n ( G y ) is the Rademacher complexity [5] w.r.t.
G y on the distribution with density p ( x ) thatoften decays in the rate of O ( 1 √ n ) .
We prove this conclusion in Appendix C. From the theorem above, we can learn that with theidentifiable condition which is a common assumption with the use of complex models [4, 25, 31],
R L Φ c (ˆ g ) converges to R ∗ L Φ c in O p (1 / √ n ) , which is the optimal parametric convergence rate withoutadditional assumptions [33].
According to Theorem 2, it is straightforward that ˜ R Φ ( g ) P
→
˜
R ∗ Φ alsoholds.
Nevertheless, the relation between the minimization of surrogate risk ˜ R Φ ( g ) and that of thetarget risk ˜ R 01 ( φ ◦ g ) is still unknown.
According to Theorem 1, the minimization of ˜ R 01 ( φ ◦ g ) isequivalent to zero-one- c risk minimization, which is the goal of CwR. We answer this question in thenext section by giving a sufficient condition for the ℓ 01 c -consistency for L Φ c .
Theoretical Analysis
In this section, we first point out the sufficient condition for L Φ c to be ℓ 01 c -calibrated and show that itis also necessary if the candidates of Φ are permutation-invariant, which consists of a large number ofcommonly used loss functions, including but not limited to cross-entropy loss, focal loss, and meanabsolute error.
Then we further specify the regret transfer bounds for a family of CPE-free surrogates[57], which has not been provided with theoretical analysis before.
5.1 Necessary and Sufficient Conditions for ℓ 01 c -Consistency
Given the loss formulation (4), a natural idea is to construct surrogate L Φ c with commonly usedmulti-class loss functions.
Here, we justify this idea by showing that we can borrow the calibrationanalyses of multi-class surrogates and set Φ to any ( K +1) -class
ℓ 01 -calibrated surrogates accordingto the following conditions:
Theorem 4.
L Φ c is ℓ 01 c -consistent for any c ∈ [0 , 1] if Φ is an ℓ 01 -calibrated surrogate loss.
Furthermore, if Φ is permutation-invariant, i.e., Φ( P g ) =
P Φ( g ) for all permutation matrices P , Φ ’sℓ 01 -calibration is also necessary for the ℓ 01 c -consistency of L Φ c .
The complete proof is shown in Appendix D and here we provide its sketch.
The equivalence between
CwR and multi-class classification on D ® c shown in Theorem 1, 2, and Corollary 1 directly yieldsthe sufficiency of this condition.
Though the equivalent classification problem is limited on D ® c ,the permutation-invariance of Φ and the arbitrariness of c require that Φ should be capable to yieldoptimal minima, which indicates the necessity of Φ ’s calibration.
As a result, we can use any Φ in an off-the-shelf manner, i.e., to the consistency of different L Φ c , weonly have to check if Φ is ℓ 01 -calibrated, which has been studied thoroughly [7, 47, 41], instead oftedious case-based discussions.
It is noticeable that when considering multi-class surrogates that arenot permutation-invariant, the calibration of Φ may not be necessary, which indicates that surrogatesthat are not calibrated and not permutation-invariant may also be potential options. Nevertheless, a classification-calibrated surrogate Φ can always be a safe choice given the sufficiency in Theorem 4and fruitful calibration analyses.
5.2 Calibration Result for Generalized Cross Entropy Loss
Given the sufficient condition for ℓ 01 c -consistency, we can construct L Φ c with any ℓ 01 -calibratedsurrogates.
However, it has been shown in Charoenphakdee et al.
[8] that it can lead to a model thatrejects more data than necessary if the cross entropy (CE) loss is used as Φ , which is a popular choiceas a surrogate.
Another common surrogate is the mean absolute error (MAE).
Though it can avoid CPE and only focus on the crucial class with the maximum posterior probability, it usually takesmore training epochs before convergence [57], which can be costly in practical use.
Here, we consider the generalized cross entropy (GCE) loss [57] that can take the advantages of the
CE loss and MAE, which is defined as below:
Definition 6.
(Generalized cross entropy losses)
For any γ ∈ (0 , 1] , the GCE loss is defined as below:
[Equation]
where S ( · ) is the softmax-transformation.
It can be seen that the loss formulation is equivalent to MAE if γ = 1 and it is also reported in Zhangand Sabuncu [57] that the GCE loss can approximate the CE loss if γ → 0 .
Though the GCE loss hasproved to be effective in practical use, to the best of our knowledge, its calibration results remainunknown. To justify the combination of GCE loss and L Φ c , we give the calibration analysis andfurther show its analytical solution.
Theorem 5.
The GCE loss Φ γ is ℓ 01 -calibrated for any γ ∈ (0 , 1] .
For the optimal modelg ∗ ,
[Equation]
for all the x ∈ X almost surely if γ ∈ (0 , 1) .
If γ = 1 ,
[Equation]
y η y
= 1 .
The proof can be found in Appendix E.
After verifying the calibration result of the GCE loss,we can combine it with the loss formulation L Φ c and obtain an ℓ 01 c -consistent surrogate.
We willexperimentally demonstrate its effectiveness in the next section.
Experiments
In this section, we provide the experiment results of CwR with deep models, which are evaluated bythe zero-one- c loss following the common practice [38, 8].
We also show the misclassification rateof the accepted data and the ratio of the rejected data.
Details of the setup and the experiments forinstance-dependent cost can be found in Appendix F and G, respectively.
Datasets and Models.
In the experiments, we evaluate the proposed methods and baselines on threewidely-used benchmarks Fashion-MNIST [52], SVHN [37], CIFAR-10 [28] with cost c selected from{ 0 . 05 , 0 . 06 , 0 . 07 , 0 .
10 } for Fashion-MNIST and { 0 . 05 , 0 .
25 , 0 . 30 } theother two.
We conduct data augmentation for CIFAR-10 and use the original datasets of FashionMNIST and SVHN in the experiments.
For Fashion-MNIST, we use a CNN defined in Charoenphakdee et al.
[8], and ResNet-18 and ResNet-34 [24] are used for SVHN and CIFAR-10, respectively.
Baselines.
We compare our method with state-of-the-art methods in CwR, including confidencebased cross entropy loss (CE) [38], learning to defer (DEFER) [35], and cost-sensitive learning-basedmethod with sigmoid loss (CS) [8], in which DEFER is a special case of our method that use crossentropy loss as Φ .
For CE, we also conduct the temperature scaling [23] to alleviate overconfidence.
For the proposed method, we use GCE with default parameter γ = 0 .
7 as suggested in Zhangand Sabuncu [57] and pairwise-sigmoid (Sigmoid) loss [56] to construct the surrogate L Φ c .
Weimplemented all the methods by Pytorch [39], and conducted all the experiments on NVIDIA
GeForce 3090 GPUs.
[Table]
Experimental Results.
As can be seen from the experimental results reported in Table 2, ourproposed method (i.e., either GCE or Sigmoid) significantly outperforms other compared methodsin most cases.
Obviously, for all the datasets and cost c , our GCE method outperforms the baseline
DEFER method, which indicates that CwR cannot be simply solved by the methods used for learningto defer.
It can be also seen that confidence-based CE is only comparable to the proposed methodon FMNIST with a simple CNN.
When complex models are used, the effect of overconfidence isinevitable even with the use of temperature scaling, which can be induced from the fact that CEoften rejects less data than GCE on SVHN and CIFAR-10.
Though CS is comparable to GCE on CIFAR-10 when the rejection cost is high, its performance degrades drastically when the rejection costdecreases, which shows that it is not the best choice in highly error-critical tasks.
When ResNet-18and ResNet-34 are used on SVHN and CIFAR-10 respectively, our GCE method outperforms oris comparable to all the baselines, which shows that GCE is more stable on complex models.
Ourproposed Sigmoid method performs better than most baselines and is comparable to CE with theuse of a simple CNN model, which aligns with the existing observations that pairwise losses areoften effective with simple models [49, 14].
These results show that our method can benefit from theflexibility of the choices of loss functions.
Conclusion
In this paper, we studied the problem of classification with rejection, which can refrain from makinga prediction to avoid critical misclassification.
We derived a novel formulation for CwR that canbe equipped with arbitrary loss functions while maintaining the theoretical guarantees, makingthem highly adaptive to the dataset in practical use.
First, we showed the equivalence between
K -class CwR and a ( K +1) -class
classification problem, and proposed an empirical risk minimizationformulation to solve this problem with an estimation error bound.
Then, we pointed out necessaryand sufficient conditions for the learning consistency of the surrogates constructed on our proposedformulation equipped with any classification-calibrated multi-class losses.
Finally, experimentalresults demonstrated the effectiveness of our proposed method.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix H.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix H.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See the
supplemental material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 6 and Appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Table 2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 6.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 6.
(b) Did you mention the license of the assets?
[N/A]
The used datasets are open bench-
marks.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Please refer to the supplemental materials.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]