Post-hoc Estimators for Learning to Defer to an Expert
Abstract
Many practical settings allow a classiﬁer to defer predictions to one or more costly experts .
For example, the learning to defer paradigm allows a classiﬁer to defer to a human expert, at some monetary cost.
Similarly, the adaptive inference paradigm allows a base model to defer to one or more large models, at some computational cost.
The goal in these settings is to learn classiﬁcation and deferral mechanisms to optimise a suitable accuracy-cost tradeoﬀ. To achieve this, a central issue studied in prior work is the design of a coherent loss function for both mechanisms.
In this work, we demonstrate that existing losses can underﬁt the training set when there is a non-trivial deferral cost, owing to an implicit application of a high level of label smoothing.
To resolve this, we propose two post-hoc estimators that ﬁt a deferral function on top of a base model, either by threshold correction, or by learning when the base model’s error rate exceeds the cost of deferring to the expert.
Both approaches are equipped with theoretical guarantees, and empirically yield eﬀective accuracy-cost tradeoﬀs on learning to defer and adaptive inference benchmarks.
Introduction: Learning to Defer to an Expert Model
Supervised classiﬁcation conventionally considers learning a single model with good average-case test performance.
However, many practical settings allow for choosing amongst multiple models to classify a given sample.
For example, in the learning to defer ( L2D ) paradigm [18], a base classiﬁer may be used in conjunction with a human expert .
Speciﬁcally, the classiﬁer has the option of deferring to the expert at the expense of incurring some additional cost (e.g., monetary expense).
The goal is to learn a classiﬁer and deferral function achieving a suitable accuracy-cost tradeoﬀ.
This problem has received a surge of interest [26, 19, 22, 34] given the increasing use of machine learning in high-stakes decision making.
Interestingly, similar considerations are also found in the problem of adaptive inference for resource-constrained prediction [13]: here, a base model can choose to defer to one or more experts, which are themselves learning models with higher capacity and inference cost.
Similar to L2D, the goal is to achieve a suitable tradeoﬀ between accuracy and (inference) cost.
A key problem in such settings is deciding when the base model should defer its prediction to the expert(s).
An intuitive strategy is to defer when the model has low prediction conﬁdence (e.g., classiﬁcation margin).
While theoretically grounded [8], this strategy can under-perform when the base model has low capacity [9].
An alternate strategy is to explicitly formulate and optimise a joint loss function for the base classiﬁer and deferral function.
This has been followed with success in the
L2D literature, particularly with the cost-sensitive softmax cross-entropy loss ( CSS ) of Mozannar and
Sontag [19], and one-versus-all ( OvA ) loss of Verma and Nalisnick [34].
[Figure]
While the above losses have proven eﬀective, they are not without limitations.
Indeed, both focus on the setting where the cost of deferring is only given by the probability that the expert makes a mistake on a sample; this does not consider the additional ﬁxed cost of querying an expert in the ﬁrst place.
As we shall see (§3), incorporating any non-zero ﬁxed cost makes these losses prone to underﬁtting ; cf.
Figure 1.
This behaviour is owing to the fact that the losses implicitly apply a high level of label smoothing .
The setting of non-zero ﬁxed cost is particularly important in adaptive inference settings, where querying an expert inherently involves paying an increased computational cost.
In this work, we resolve this issue via deferral schemes based on post-hoc estimators .
In a nutshell, we follow a two-step procedure wherein we ﬁt a deferral function on top of a base model, either by threshold correction, or by learning to predict when the base model’s error rate exceeds the deferral cost.
This allows us to build on top of the standard (zero ﬁxed cost) CSS and OvA solutions, resulting in an empirically eﬀective strategy with theoretical guarantees.
In sum, our contributions are:
(i) we identify limitations in prior losses for L2D, particularly when the cost of deferring to an expert involves a non-zero ﬁxed cost in addition to the probability of the expert making a mistake (§3) (ii) we propose new post-hoc estimators for learning a deferral function, which overcome the above limitation while enjoying theoretical guarantees (§4, Theorem 2, Theorem 4) (iii) we empirically show that the new post-hoc estimators work well on synthetic and real-world benchmark datasets for both L2D and adaptive inference settings (§5).
In the course of our analysis, we additionally generalise the OvA loss to reliably account for general deferral costs ((10), Lemma 1), and to reliably estimate label probabilities in practical settings (12).
Background and Notation
Our interest is in learning settings where a classiﬁer is equipped with the option of deferring to an expert model.
For concreteness, we shall focus our exposition on the learning to defer paradigm.
2.1 Multi-class Classiﬁcation
Fix an instance space X and label space Y =
[ L ] .
= { 1 , 2 , . . .
, L } , and let P be a distribution over
X × Y .
Given a training sample S =
{ ( x n , y n ) } n ∈ [ N ] drawn from P , multi-class classiﬁcation seeks a classiﬁer h : X → Y with low misclassiﬁcation error R ( h ) =
P ( y (cid:54) =
h ( x )) .
We may parameterise h as h ( x ) = argmax y (cid:48)
∈
Y f y (cid:48)
( x ) for a scorer f : X → R L .
Such a scorer may be learned by minimising the empirical surrogate risk
ˆ R ( f )
= 1 N (cid:80) n ∈
[ N ] (cid:96) ( x n , y n , f ( x n )) for some loss (cid:96) :
X × Y × R L → R + , such as the softmax cross-entropy (cid:96) ( x, y, f ( x ))
=
log (cid:104)(cid:80) y (cid:48)
∈
Y e f y (cid:48)
( x ) (cid:105) − f y ( x ) .
For binary classiﬁcation problems with L = 2 , it is common to enforce f 2 ( x ) = − f 1 ( x ) and set (cid:96) ( x, y, f ( x )) =
φ ( f y ( x )) for some margin loss φ : R → R + , such as φ ( z )
=
log(1 + e − z ) .
2.2 Learning to Defer (L2D) Problem
In the learning to defer ( L2D ) problem [18], one seeks a classiﬁer that can either make a standard prediction in Y via some base model, or defer its prediction to an expert model h exp : X → Y , at the expense of incurring a sample-dependent cost c exp ( x, y ) > 0 .
Typically, c exp takes into account both the ﬁxed cost of deferring to the expert, which we will assume to be a constant c 0 ∈ [0 , 1] (e.g., normalised monetary cost), and an estimate of the probability that the expert makes a mistake on the example.
Canonical examples include c (1) exp ( x, y ) =
c 0 and c (2) exp ( x, y )
=
c 0 + 1 ( y (cid:54) =
h exp ( x )) .
The choice c (1) exp ( x, y ) corresponds to learning with a reject option or abstention [8, 2, 9, 6].
Given a suitable c exp ( x, y ) , the learning problem may be formalised as follows.
Consider a classiﬁer ¯ h : X → Y ∪ {⊥} equipped with a “defer” option ⊥ .
Our goal is to minimise
[Equation]
Intuitively, when ¯ h chooses not to defer, we incur the usual misclassiﬁcation error; otherwise, we incur the cost c exp ( x, y ) of invoking the expert h exp .
One can cast (1) as an instantiation of cost-sensitive learning [11]; see Appendix B. We remark that Okati et al.
[22] considered a slightly diﬀerent setting, wherein one enforces a hard constraint on the expert cost c exp ( x, y ) .
We focus on a soft additive penalty in (1), but we may handle hard constraints via standard Lagrangian theory; see Appendix G.
In practice, it is useful to parameterise ¯ h in terms of a scorer ¯ f : X → R L +1 with label logits { ¯ f 1 ( x ) , . . .
, ¯ f L ( x ) } , and “defer” logit
¯ f ⊥ ( x ) .
We then predict via ¯ h (
[Equation]
argmax
[Equation]
In practice, we may parameterise ¯ f via neural network logits: for any y (cid:48)
∈ Y ∪{⊥} , ¯ f y (cid:48)
( x ) = w (cid:62) y (cid:48)
Φ( x ) for weights w y (cid:48) and embedding Φ .
This parameterisation induces implicit sharing amongst all logits.
One conceptually useful quantity is the Bayes-optimal ¯ h ∗ =
argmin ¯ h : X → Y ∪{⊥} R def (¯ h ) , which is
[Equation]
Intuitively, we defer iﬀ the expected cost of querying the expert is smaller than the expected error of the base model.
For the constant cost function c (1) exp ( x, y ) , this is known as Chow’s rule [8].
2.3 Losses for Learning to Defer
We review some popular recent strategies to learn ¯ f (and thus ¯ h ) below.
Conﬁdence thresholding .
For c exp ( x, y )
=
c 0 , one strategy is to learn ¯ f 1 , . .
, ¯ f L by minimising the softmax cross-entropy, and ﬁx ¯ f ⊥ ( x ) = 1 − c 0 .
This conﬁdence thresholding approach [21] mimics the Bayes-optimal classiﬁer (3).
A related series of approaches [27, 38, 1] make deferral decisions by comparing the base model conﬁdence with the conﬁdence in the expert being correct.
When the base model has low-capacity, such approaches may be sub-optimal as the base model may sub-optimally allocate its ﬁnite capacity by classifying examples that the expert can easily predict.
Cost-sensitive softmax cross-entropy ( CSS ) .
To overcome the limitations of conﬁdence thresholding, it is desirable to explicitly learn ¯ f ⊥ as well.
To this end, Mozannar and Sontag [19] proposed a cost-sensitive softmax cross-entropy ( CSS ): for constant upper bound c max ≥ c ( x, y ) , this is
[Equation]
where ¯ p y ( x ) = exp(
¯ f y ( x )) / (cid:80) y (cid:48)
∈ Y ∪{⊥} exp( ¯ f y (cid:48)
( x )) is the softmax distribution over the standard and the defer label.
For c exp ( x, y )
=
c 0 + 1 ( y (cid:54) =
h exp ( x )) , we have c max = 1 + c 0 .
The loss (cid:96) CSS is
Fisher-consistent [19, Proposition 1], with Bayes-optimal prediction
[Equation]
One-versus-all ( OvA ) .
Despite the Fisher consistency of (cid:96) CSS , Verma and Nalisnick [34] observed that empirically, the loss may not produce calibrated estimates of the expert’s error probability.
Verma and
Nalisnick thus proposed a one-versus-all ( OvA ) loss for the cost function c exp ( x, y )
= 1 ( y (cid:54)
=
h exp ( x )) :
[Equation]
OVA ( x, y, ¯ f
[Equation]
=
[Equation]
Here, φ : R → R + is a binary proper composite loss [28], e.g., φ ( z )
=
log(1 + e − z ) .
Such losses have an associated inverse link function ψ : R → [0 , 1] , and produce Bayes-optimal scorer
[Equation]
Intuitively, both (4) and (6) seek to have the ﬁrst L logits { ¯ f 1 ( x ) , . .
, ¯ f L ( x ) } estimate P ( y | x ) , and the reject logit ¯ f ⊥ ( x ) model the cost of querying the expert.
With this, the highest scoring logit provides us with an estimate of the Bayes-optimal deferral decision (3).
Limitations of Existing Learning to Defer Losses
We now study existing losses for learning to defer more carefully, and show that they may underﬁt in an important practical setting.
Speciﬁcally, recall that a canonical choice of deferral cost function in (1) is c exp ( x, y )
= 1 ( y (cid:54)
=
h exp ( x ))
+ c 0 , for constant c 0 ≥ 0 .
Here, c 0 represents a ﬁxed cost of querying the expert, which is independent of whether the expert misclassiﬁes the sample.
Thesetting of non-zeroﬁxedcost c 0 > 0 isof practicalinterest; e.g., thisisintrinsictoadaptiveinference settings, wherein querying the expert model fundamentally involves increasing a computational overhead (see §4.3 for more discussion).
Thus, an ideal surrogate loss for (1) should be performant for c 0 > 0 .
Unfortunately, we now show that this does not hold for the CSS and OvA losses.
3.1 Underﬁtting of Cost-Sensitive Softmax Cross-Entropy when c 0 >
The cost-sensitive softmax cross-entropy ( CSS ) is in principle applicable for any deferral cost c exp ( x, y ) , and in particular c exp ( x, y )
=
c 0 + 1 ( y (cid:54) =
h exp ( x )) for any c 0 ≥ 0 .
However, the focus in Mozannar and Sontag [19] was the case c 0 = 0 .
In fact, we now show that when c 0 > 0 , the loss can underﬁt .
To see this, note that the tightest possible maximal cost is c max = 1 + c 0 .
With this choice, (4) becomes
[Equation]
When c 0 = 0 , the second term vanishes, and we have a conic combination of the log-loss on the true and deferral labels.
When c 0 > 0 , the second term is active, and applies a form of label smoothing [31]: we allow for all labels y (cid:48)
∈ Y − { y } to be potential “positives” for x , with a weight of c 0 ∈ [0 , 1] .
Put diﬀerently, we treat the ground-truth label distribution as a mixture of a one-hot distribution on y , and a uniform distribution over all labels.
As c 0 → 1 − , we apply a large amount of smoothing on labels other than y ; this can be problematic.
Concretely, consider an x where the expert gives the wrong prediction, i.e., P ( y = h exp ( x ))
= 0 .
Then, the Bayes-optimal solution (5) will be
[Equation]
Observe that for c 0 > 0 , as L increases, the range of values in ¯ p ∗ shrinks: indeed, the gap between the optimal probability for true label y and a competing label y (cid:48) will be O (1 /L ) .
This small probability margin makes learning challenging: given a ﬁnite sample, it will be easy for the model to mistakenly swap the true label y with some competing label y (cid:48) .
Thus, in settings where L is moderately large, and where there is a non-zero ﬁxed deferral cost c 0 > 0 , (cid:96) CSS may underperform.
Such issues have also been observed [7] for a similar loss developed for label noise [25].
As an illustration, we consider in Figure 1 a learning to defer setting comprising a ResNet-8 base model and a ResNet-32 “expert” on CIFAR-100.
We assume a deferral cost c exp ( x, y ) =
c 0 + 1 ( y (cid:54) =
h exp ( x )) , where c 0 is varied in [0 , 1] .
For each c 0 , we train the base model using (4), and report the train and test accuracy.
As c 0 increases, the base model shows signiﬁcant training accuracy degradation, i.e., it underﬁts.
The high level of label smoothing plays a role: in the right panel, we see that the entropy of the model’s predicted distribution over labels increases with c 0 .
(The entropy is normalised by log L to lie in [0 , 1] .)
See Appendix I for additional plots, and Appendix I.3 for results when c exp ( x, y )
=
c 0 .
3.2 Underﬁtting of One-Versus-All Loss when c 0 >
The one-versus-all ( OvA ) loss of Verma and Nalisnick [34] was presented for the speciﬁc case of c 0 = 0 ; as stated, the loss is not applicable when c 0 > 0 .
We may however extend this loss to a general c exp ( x, y ) with upper bound c max ≥ 1 (see Appendix D for a more general derivation):
[Equation]
Compared to (6), we have an additional weighted sum over each φ y (cid:48)
( x ) .
As with OvA , this loss results in a coherent Bayes-optimal solution when φ is a strictly proper composite loss [5, 28].
Recall that such losses are characterised by an inverse link function ψ : R → [0 , 1] , such as the logistic loss φ ( z ) =
log(1 + e − z ) with sigmoid inverse link ψ ( v ) =
(1 + exp( − v )) − 1 .
Lemma 1.
Suppose φ is a strictly proper composite loss with inverse link function ψ .
Then, the one-versus-all loss in (10) is calibrated for the learning to defer risk (1) , with Bayes-optimal scorer
[Equation]
] c max .
Unlike the optimal scorer for the CSS loss, here (cid:80) y (cid:48) ∈
[ L ] ψ ( ¯ f ∗ y (cid:48)
( x )) (cid:54) = 1 .
Nonetheless, the highest scoring logit agrees with the Bayes-optimal classiﬁer (3) for learning to defer.
Reviewing (10), we see that when c max > 1 , we apply a form of label smoothing by additional terms of the form (cid:80)
[Equation]
x )) , i.e., we eﬀectively treat all y (cid:48) (cid:54) =
y as potential positives.
As with the cost-sensitive softmax entropy, this can result in underﬁtting; recall that with c exp ( x, y )
=
c 0 + 1 ( y (cid:54) =
h exp ( x )) , we have an upper bound c max = 1 + c 0 > 1 .
We empirically conﬁrm in Appendix I.2 that we indeed observe analogous underﬁtting behaviour as per the CSS loss in Figure 1.
The above shows that the CSS and OvA losses have a subtle limitation when c 0 > 0 .
This is not in conﬂict with the Fisher-consistency of these losses: the latter establishes that the asymptotic minimiser of these losses agrees with the Bayes-optimal classiﬁer (3).
Our analysis explicates that the precise form of these Bayes-optimal solutions may not be amenable to learning from a ﬁnite sample .
Since both losses seamlessly handle c 0 = 0 , a natural question is whether one can leverage this solution to help guide the one for c 0 > 0 .
We now present post-approaches that do precisely this.
Post-hoc Approaches to Learning to Defer
We now present two simple post-hoc strategies for learning to defer, which build on the CSS and OvA solutions for c 0 = 0 via threshold correction and re-training the deferral function.
4.1 Post-hoc Threshold Correction
Our ﬁrst post-hoc strategy (Figure 2) exploits the form of the Bayes-optimal classiﬁer in (3): to mimic this classiﬁer for the cost function c exp ( x, y )
= 1 ( y (cid:54)
=
h exp ( x ))
+ c 0 , it suﬃces to obtain estimates of P ( y | x ) and the expected deferral cost E [ c exp ( x, y )]
=
P ( y (cid:54) =
h exp ( x ))
+ c 0 .
Consequently, suppose we learn models π : X → [0 , 1] L and (cid:15) :
X → [0 , 1] estimating P ( y | x ) and P ( y (cid:54) =
h exp ( x )) respectively.
Then, for any c 0 ≥ 0 , we can approximate (3) via the classiﬁer
[Equation]
Intuitively, this procedure ought to perform well when π ( x ) is close to η ( x ) .
=
[ P ( y | x )] y ∈ Y , and (cid:15) ( x ) is close to P ( y (cid:54) =
h exp ( x )) .
The following excess risk bound for ¯ h formalises this intuition.
Theorem 2.
Pick any π : X → [0 , 1] L and (cid:15) :
X → [0 , 1] .
Let ¯ h be the corresponding classiﬁer in (11) .
Then, the excess risk of ¯ h compared to the Bayes-optimal classiﬁer ¯ h ∗ (3) is:
[Equation]
Trainingsample
Minimise SOvA loss (12) with c 0 =
Base model training π y ( x ) =
exp(¯ f y
x ))
P y 0 ∈
Y exp(¯ f y
( x )) (cid:15) ( x ) =
ψ (¯ f ⊥ ( x ))
Compute statistics
Classify via (11) for any c
Final classiﬁer { ( x n ,y n ) } Nn =
Logits ¯ f 1 ,..., ¯ f L
Deferlogit ¯ f ⊥ π ( x ) ,(cid:15) ( x )
Crucially, observe that we can learn π and (cid:15) once , and then sweep across diﬀerent c 0 values in (11).
Thus, we can attempt to leverage the solutions for either the CSS or OvA losses with c 0 = 0 — recalling that these do not exhibit underﬁtting, unlike when c 0 > 0 — and then simply vary c 0 post-hoc.
Indeed, both losses’ Bayes-optimal solutions (5), (7) for c 0 = 0 suggest that a transform of ¯ f 1 , . . .
, ¯ f L provide estimates of P ( y | x ) , while a transform of ¯ f ⊥ provides an estimate of P ( y (cid:54)
=
h exp ( x )) .
Unfortunately, in practice, both losses are potentially suboptimal to learn π, (cid:15) .
For the CSS loss, Verma and Nalisnick [34] showed it does not provide calibrated estimates of P ( y (cid:54)
=
h exp ( x )) .
On the other hand, while the OvA loss provides calibrated estimates of P ( y (cid:54) =
h exp ( x )) , its probability estimates of P ( y | x ) may underperform when L is large (see Appendix J).
We overcome this with a simple solution: we estimate π via the softmax cross-entropy, and (cid:15) via the OvA loss.
This is achieved via a hybrid softmax cross-entropy plus OvA loss ( SOvA ): (cid:96) SOVA ( x, y,
[Equation]
=
[Equation]
x )) , (12) where p y ( x ) = exp( ¯ f y ( x )) / (cid:80) y
[Equation]
Y exp( ¯ f y
[Equation]
( x )) .
Following the Bayes-optimal solution, we set π ( x ) =
( p 1 ( x ) , . . .
, p L ( x )) and (cid:15) ( x ) = ψ ( ¯ f ⊥ ( x )) .
As with the CSS and OvA losses, we may parameterise ¯ f y (cid:48)
( x ) = w (cid:62) y (cid:48)
Φ( x ) for shared Φ , and thus induce information sharing amongst all logits.
The above post-hoc approach crucially relies on estimating the expert’s error rate.
A potentially simpler task is learning to predict whether it is beneﬁcial to invoke the expert, i.e., whether the expert’s error is lower than that of the base model.
We now consider an alternate strategy that implements this.
4.2 Post-hoc Rejector Training
Our second post-hoc strategy revisits the fundamental risk (1) underpinning the learning to defer problem.
While deﬁned in terms of a classiﬁer ¯ h : X → Y ∪ {⊥} , we may rewrite this in terms of a base classiﬁer h : X → Y and rejector r : X → R , with the latter denoting the conﬁdence in deferring:
[Equation]
(13) where c mod ( x, y )
= 1 ( y (cid:54) =
h ( x )) .
Given a scorer ¯ f : X → R L +1 , we may set h ( x ) = argmax y (cid:48)
∈ Y ¯ f y (cid:48)
( x ) and r ( x ) =
¯ f ⊥ ( x ) − max y (cid:48)
∈ Y ¯ f y (cid:48)
( x ) .
Both CSS and OvA construct surrogate losses that are applied jointly over all logits ¯ f 1 , . .
, ¯ f L , ¯ f ⊥ ; equally, they construct a joint surrogate loss for both the base classiﬁer h and rejector r .
As established earlier, the solution obtained from these losses is reasonable under ﬁxed deferral cost c 0 = 0 , but tends to suﬀer when c 0 > 0 .
We may resolve this issue via a two-step procedure (see Figure 3): in the ﬁrst step, we obtain standard logits ¯ f 1 ( x ) , . .
, ¯ f L ( x ) by minimising either the CSS or OvA loss with c 0 = 0 .
In the second step, we train only the deferral logit, by constructing a partial surrogate for only the rejector r ( x ) in (13).
Concretely, for a binary surrogate loss φ : R → R + , we may deﬁne the following surrogate for r :
[Equation] [Equation]
and minimize the resulting partial surrogate risk over r :
[Equation]
With this loss, c 0 only plays a role in the c exp ( x, y ) weighting term, and thus only inﬂuences the deferral decision; it does not induce any underﬁtting of the standard logits ¯ f 1 , . .
, ¯ f L .
Observe also that c mod ( x, y ) will involve the frozen logits ¯ f 1 , . .
, ¯ f L obtained from the ﬁrst stage.
We may further leverage these logits by parameterising r ( x ) =
¯ f ⊥ ( x ) − max y (cid:48)
∈ Y ¯ f y (cid:48) ( x ) , and optimising only the ¯ f ⊥ ( x ) term.
In practice, exploiting the probabilities from the ﬁrst stage model can oﬀer a closer approximation to the Bayes-optimal classiﬁer; see Appendix F for more discussion.
Theoretically, the surrogate loss in (14) provides a calibrated deferral rule for any base classiﬁer h .
For strictly proper composite φ with inverse link ψ : R → [0 , 1] , the optimal rejector is as follows.
[Figure]
Lemma 3.
The Bayes-optimal rejector for the loss (14) is ψ ( r ∗ ( x ))
=
[Figure]
x
[Figure]
y
[Figure]
Note that when the ﬁrst stage logits agree with the Bayes-optimal solution, E y | x
[ c mod ( x, y )] = 1 − max y ∈ Y P ( y | x ) , and so deferring when ψ ( r ∗ ( x )) > 0 .
5 produces the optimal decision.
We next provide a bound an excess risk bound for a classiﬁer learned using the post-hoc training procedure, under the assumption that the surrogate loss φ used is classiﬁcation calibrated [3].
Theorem 4.
Suppose φ is classiﬁcation calibrated and c exp (
x, y ) ∈
[ c min , c max ] for some c min > 0 .
Denote the misclassiﬁcation risk for classiﬁer h by R err ( h ) =
E [ 1 ( y (cid:54) =
h ( x ))] .
Let ˆ r be the rejector obtained by minimizing the partial surrogate risk in (15) for base classiﬁer ˆ h .
Then the excess L2D risk for the resulting classiﬁer ( ˆ h, ˆ r ) is bounded in terms of the excess surrogate risk for the rejector ˆ r and the excess misclassiﬁcation risk for the base classiﬁer ˆ h :
[Equation]
for some increasing function Ψ : R + → R + with Ψ(0) = 0 .
4.3 Discussion and Extensions
Relation to existing work .
Post-hoc threshold correction generalises Chow’s rule [8] for learning to reject, wherein c exp ( x, y )
=
c 0 .
It also generalises the approaches of Raghu et al.
[27], Wilder et al.
[37], Bansal et al.
[1] for learning to defer: these assume c 0 = 0 , and use separate models to estimate π ( x ) and (cid:15) ( x ) .
By contrast, we estimate these quantities with a single neural model, by minimising the SOvA loss in (12) with ¯ f y (cid:48)
( x ) = w (cid:62) y (cid:48)
Φ( x ) for y (cid:48) ∈ Y ∪ {⊥} .
As with the standard OvA loss, this induces implicit information sharing between the defer logit ¯ f ⊥ and the standard logits ¯ f 1 , . .
, ¯ f L ; thus, the estimates of the expert error probability will be inﬂuenced by the base model.
Note that the conﬁdence approach of Raghu et al.
[27] was shown to be outperformed by the CSS and OvA losses in Mozannar and Sontag [19], Verma and Nalisnick [34]; our proposed threshold correction further achieves competitive or superior performance to these losses (§5).
Verma and Nalisnick [34, Appendix E] considered a post-hoc scheme for the setting where there is a hard constraint on the expert cost.
Here, one trains a probabilistic model ¯ p using the OvA loss, and defers on the samples with the largest values of ¯ p ⊥ ( x ) − max y (cid:48)
∈ Y ¯ p y (cid:48) ( x ) .
While derived for a diﬀerent setting, this is similar to (11), as varying the cost c 0 eﬀectively changes the fraction of samples that are deferred.
Note however that we use the SOvA rather than OvA loss to obtain π, (cid:15) , which can have a notable impact on performance when L is large (see Appendix J).
Cortes et al.
[9] proposed a joint surrogate over r, ¯ f for (13) in the special case of constant c exp ( x, y )
=
c 0 , and a scalar scorer ¯ f : X → R for binary labels.
By contrast, the surrogate loss (14) handles multi-class ¯ f , generic cost functions c exp , and provides a partial surrogate over only r .
Adaptive inference and multiple experts .
Post-hoc training is naturally amenable to settings where there are multiple experts.
Concretely, suppose there are K experts with logits ¯ f (1) ( x ) , . .
, ¯ f ( K ) ( x ) , and our goal is to learn which expert to employ for a given sample.
One may learn multiple deferral functions ¯ f (1) ⊥ ( x ) , . . .
, ¯ f ( K ) ⊥
( x ) in a post-hoc manner, and choose the smallest k with ¯ f ( k ) ⊥ ( x ) < 0 .
This setting arises naturally in adaptive inference [13], where our goal is to reduce the inference cost of a learning system by allowing for variable compute across samples: intuitively, we would like to spend less computation on “easy” samples.
To achieve this, one may construct several models (“experts”) of increasing inference cost, and defer to a suitable expert.
Such models include early-exit
[Figure]
classiﬁcation heads at intermediate nodes of a network [24, 32, 36, 15, 29], or sequential cascades of models of increasing complexity [4, 30, 35].
Conceptually, adaptive inference involves the same core problem as learning to defer: at any given intermediate model, one has the option to defer prediction to the next model, at the expense of increasing the overall computation cost.
Interestingly, similar ideas to post-hoc training have been explored in adaptive inference [33].
Compared to this work, we present a uniﬁed view of the learning to defer and adaptive inference problems, demonstrate limitations of existing losses for the former, and leverage generic logits ¯ f 1 , . .
, ¯ f L from any base classiﬁer.
Experimental Results
We now present empirical results illustrating the eﬃcacy of both our proposed posthoc estimators.
5.1 Results on Synthetic Data
We begin with a synthetic problem where X ⊂ R 2 .
We consider a distribution P ( x, y )
= (cid:80) a ∈{ 0 , 1 } P ( x, y, a ) , where a ∈ { 0 , 1 } denotes some latent subgroup.
Figure 4 shows a sample draw from P ( x, y ) : in a nutshell, P ( x, y | a = 0) is a mixture of isotropic Gaussians, while P ( x, y | a = 1) comprises two concentric circles.
See Appendix H.1 for a precise speciﬁcation.
We consider learners that have access to a sample { ( x n , y n ) }
Nn =1 from P ( x, y ) , and to expert model predictions { h exp ( x n ) } Nn =1 .
The expert is taken to be a linear classiﬁer that is trained on only samples with a = 1 , using quadratic features: i.e., for x = ( x 1 , x 2 ) , we construct Φ( x ) =
( x 1 , x 2 , x 21 , x 22 , x 1 · x 2 ) .
We set the expert deferral cost function to be c exp ( x, y ) =
c 0 + 1 ( y (cid:54) =
h exp ( x )) .
We train linear classiﬁers using conﬁdence thresholding based on a constant deferral cost c 0 ( Chow ), the CSS and OvA losses, and our proposed post-hoc approaches.
For each learner, we vary the ﬁxed deferral cost c 0 from { 0 . 0 , 0 . 05 , 0 . 10 , . . .
For each setting, we compute the fraction of samples deferred to the expert, and the overall accuracy of the base model plus expert.
We conduct 250 independent trials, each with a diﬀerent draw of the training sample and expert predictions.
We highlight a few trends from the results in Figure 4.
First, Chow strongly underperforms post-hoc thresholding when many samples are deferred.
This is because the latter does not consider the expert’s error (which is high on a = 1 ) when deferring.
Second, post-hoc training is competitive or superior to
CSS and OvA at most operating points.
Third, post-hoc thresholding underperforms post-hoc training.
This is by construction, as it is challenging to reliably estimate P ( y | x ) with a linear model.
However, we shall now see that on real-world benchmarks, thresholding is far more competitive.
5.2 Results on Real-World Data
We now report results on the CIFAR-10, CIFAR-100 [16], and ImageNet [10] datasets.
Learning to defer .
Inspired by Mozannar and Sontag [19, Section 6.2], we consider a setting where there is a “specialist” expert: we train an expert model on only samples with label belonging to some subset Y sub .
Intuitively, this model “specialises” to only samples from Y sub .
We then consider a base model that is trained on all samples, but is allowed to defer to the “specialist” expert.
On
[Figure]
CIFAR-10 and CIFAR-100, we use a ResNet-8 base model and ResNet-56 expert; on ImageNet, we use a MobileNet-v2 base model and an EﬃcientNet-B0 expert.
The label subset Y sub comprises the ﬁrst 5 labels on CIFAR-10, the 50 labels corresponding to the ﬁrst 10 “coarse labels” on CIFAR-100, and all labels corresponding to the “dog” synset on ImageNet.
We consider a cost c exp ( x, y )
=
c 0 + 1 ( y (cid:54) =
h exp ( x )) of deferring to the expert model.
Per Figuresand 3, we apply posthoc threshold correction on top of the SOvA solution with c 0 = 0 , and posthoc training on top of the CSS solution with c 0 = 0 .
Figure 5 compares all approaches as we vary the ﬁxed cost c 0 , reporting the fraction of samples deferred to the expert, and the overall accuracy of the base model plus expert.
Our posthoc estimators oﬀer gains over the existing approaches at most operating points.
We particularly see gains in the regime where only a small fraction of samples are deferred to the expert, which corresponds to larger values of c 0 .
Between the two post-hoc methods, we see that the threshold correction approach tends to have a (slight) edge.
This suggests that on these benchmarks, it is possible to obtain suﬃciently reliable estimates of P ( y | x ) and P ( y (cid:54) =
h exp ( x )) .
We reiterate that when c 0 is large, the existing CSS and OvA losses tend to degrade owing to underﬁtting.
Indeed, their performance in this regime is signiﬁcantly lower than simply using the student model trained with the softmax cross-entropy (corresponding to Chow with 0% of samples deferred).
Note also that in comparison to the synthetic dataset, here L is larger and thus further exacerbates underﬁtting.
Finally, as in the synthetic dataset, Chow strongly underperforms when a large fraction of samples are deferred to the expert; this is because it does not take into account the expert’s error rate, and thus potentially defers when the expert performs akin to random guessing.
Adaptive inference .
We conclude with results on the related problem of adaptive inference.
We consider a setting where there is a cascade of two models of diﬀering inference costs, as measured in ﬂoating point operations ( FLOPs ), and we seek to learn a deferral function that intelligently forwards a small subset of samples to the latter model.
This allows for an inference scheme with a favourable accuracy-compute tradeoﬀ. Compared to the previous section, we now train the experts on all labels.
We report results on CIFAR-100 and ImageNet, with the former using ResNet-8 and ResNetmodels, and the latter using a MobileNet-v2 and EﬃcientNet model.
As before, we consider the deferral cost function c exp ( x, y ) =
c 0 + 1 ( y (cid:54) =
h exp ( x )) .
Figure 6 compares the FLOPs versus the overall accuracy of the base model plus expert.
Here, the CSS and OvA methods again degrade in the low FLOPs (high c 0 ) regime.
However, the Chow baseline is more competitive, since the expert models tend to be highly accurate on all samples.
Our posthoc estimators are seen to strongly outperform CSS and OvA , while being competitive with or slightly superior to Chow .
Discussion and Future Work
While post-hoc training resolves issues identiﬁed with previous joint surrogate losses, it is not without limitation: it relies crucially on reliable base classiﬁers from the ﬁrst phase.
As formalised in
Theorem 2, any degradation in performance for the latter is also transferred to the ﬁnal post-hoc predictor.
As this may be further compounded with estimation errors in the second phase, it is natural to ask whether there may be other joint surrogates that do not exhibit underﬁtting.
One
[Figure] [Figure]
u r a c y ( % )
[Figure] [Figure]
cc u r a c y ( % ) 6.
[Figure]
[Figure] [Figure]
FLOPs
[Figure] [Figure]
cc u r a c y ( % )
Chow
[Figure]
Post
[Figure]
Training
Post
[Figure]
candidate is the cost-sensitive loss of Lee et al.
[17], which is Fisher-consistent and is suitable when c 0 > 0 .
This loss imposes the constraint that
[Equation]
, and takes the form
[Equation]
( x, y, ¯ f ( x ))
=
[Equation]
One may show that under this loss, the gap between the Bayes-optial scores for the highest scoring label and any competing label is O (cid:0)
[Equation]
+
[Equation]
L (cid:1) , and thus is not adversely aﬀected as L increases.
As a qualifying comment, the loss poses challenges due to being non-diﬀerentiable, and requiring constraints on the logits.
Nonetheless, further study of the viability of this loss would be worthwhile.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The paper is primarily analytical in nature: it identiﬁes mathematical and empirical limitations of existing methods, and proposes new methods for a problem where a model can query an “expert”s.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] The newly proposed methods in this paper involve simple post-hoc operations over existing models.
Thus, they do not add signiﬁcant computational overhead.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or oﬀensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Supplementary material
for “Post
-hoc estimators for learning to defer to an expert”
A Proofs A.1 Proof of Lemma
Proof.
To show that our proposed loss is calibrated for the cost function c exp ( x, y ) , we write out the conditional risk for the loss:
[Equation]
For any y ∈ Y , the optimal scorer is thus
[Equation]
Similarly, for the defer logit,
[Equation]
By deﬁnition of a strictly proper composite loss [28], the optimal scorer f ∗ then takes the form:
[Equation]
where ψ is the inverse link function associated with φ , and
the corresponding classiﬁer ¯ h ∗ ( x ) ∈ argmax y (cid:48) ∈
[ L ] ∪{⊥} f ∗ y (cid:48)
( x ) is given by:
[Equation]
⊥ otherwise , as desired.
A.2 Proof of Theorem
Proof.
The proof is similar to that of a regret bound in Narasimhan et al.
[20, Lemma 14] for a generalcost-sensitive risk.
As noted in §4.1, we denote η y ( x ) =
P ( y | x ) and η ( x ) .
=
[ P (1 | x ) , . . .
, P ( L | x )] (cid:62) .
We will denote the expected cost of deferring to the expert using C exp ( x ) .
=
E y | x [ c exp ( x, y )] and use ˆ C exp ( x ) to denote an estimate of this quantity.
We then re-write the learning to defer risk in (1) in terms of η ( x ) and C exp ( x ) :
[Equation]
where (cid:104)· , ·(cid:105) denotes standard inner-product, e mod ˆ y
[Equation]
and e
[Equation]
We also deﬁne an empirical version of the learning to defer risk, in which η ( x ) is replaced by an estimate π ( x ) and the cost C exp ( x ) is replaced by ˆ C exp ( x ) :
[Equation]
where we denote ˆ e exp
ˆ y
( x ) =
[Equation]
Finally
[Equation]
consider an empirical version of the Bayes-optimal classiﬁer in (3):
[Equation]
⊥ else .
From the deﬁnition of ¯ h
[Equation]
we have that ¯ h ( x ) minimizes (cid:104) π ( x ) , e mod h ( x ) + ˆ e exp h
( x ) ( x ) (cid:105) for each x
[Equation]
and as a result
[Equation]
for any classiﬁer h :
[Equation]
The excess risk can then be expanded as:
[Equation]
=
[Equation]
[Table]
[Equation]
where the inequality in (17) follows from (16) with h
= h ∗ ; the inequality in (18) follows from
Hölder’s inequality; and the inequality in (19) uses the fact that (cid:107) π ( x ) (cid:107) 1 = 1 and 1 (¯ h ( x ) (cid:54) = ⊥ ) ≤ 1 .
The last step uses Hölder’s inequality.
Further expanding the inner ∞ -norm term in (20), we have for any x :
[Equation]
where we use the fact that C exp ( x ) ≤ c max and c max ≥ 1 .
Substituting this back in (20), we get:
[Equation]
Setting C exp ( x )
=
P ( y (cid:54) =
h exp ( x ))
+ c 0 and ˆ C exp ( x ) =
(cid:15) ( x ) + c 0 completes the proof. A
Proof.
The conditional risk for the loss is
[Equation]
where
[Equation]
Consequently, since φ is strictly proper composite, the Bayes-optimal must satisfy [28] ψ ( r ∗ ( x ))
=
a ( x ) .
Note that a ( x ) > 0 .
5 ⇐⇒ E y | x
[ c mod ( x, y )] > E y | x [ c exp ( x, y )] , i.e., the cost of querying the model exceeds that of querying the expert.
A.4 Proof of Theorem
Let ¯ h ∗ denote the Bayes-optimal classiﬁer that minimizes the L2D risk in (13).
We can re-write ¯ h ∗ in terms of a base classiﬁer and a rejector:
[Equation]
x
[Equation]
⊥ else.
, where h ∗ ( x ) ∈ argmax y ∈ Y
P ( y | x ) and sign ( r ∗ ( x ))
= sign
[Equation]
Notice that h ∗ is also the Bayes-optimal classiﬁer for the standard misclassiﬁcation error R err ( h ) .
We will ﬁnd it useful to ﬁrst deﬁne a normalized version of the L2D risk:
[Equation]
, where
[Equation]
[Equation]
and
[Equation]
Similarly, we deﬁne the normalized partial surrogate risk as:
[Equation]
where
[Equation]
We will also ﬁnd the following lemmas useful for proving the theorem.
Lemma 5.
Suppose c exp ( x, y ) ∈ [ c min , c max ] for some c min > 0 .
Then for any ﬁxed base classiﬁer ˆ h
[Equation]
the normalized excess risks can be bounded in terms of the unnormalized excess risks as follows:
[Equation]
Proof.
Expanding the normalized excess misclassiﬁcation risk
[Equation]
we have:
[Equation]
where in the second step
[Equation]
we use the fact that the optimal rejector is point-wise optimal
[Equation]
and in the penultimate step
[Equation]
we use Z ( x
[Equation]
=
[Equation]
Similarly, expanding the normalized excess surrogate risk
[Equation]
we have:
[Equation]
where we have used the fact Z ( x ) ≥ c min .
The next lemma builds on Lemma 5 to provide a excess risk bound for the rejector.
Lemma 6.
Suppose φ is classiﬁcation calibrated and c exp (
x, y ) ∈
[ c min , c max ] for some c min > 0 .
Then for any ﬁxed base classiﬁer
[Equation]
for some increasing function Ψ : R + → R + with Ψ(0) = 0 .
Proof.
Using the fact that φ is classiﬁcation calibrated, one can directly apply the surrogate excess risk bounds from Bartlett et al.
[3] to the normalized surrogate risk.
Speciﬁcally, we ﬁrst note that for the label distribution q ( x ) =
[Equation]
, the normalized risks can be written as: [Equation]
[Equation]
We can then bound the excess normalized surrogate risk in terms of the excess normalized misclassiﬁcation risk:
[Equation]
for some increasing function ζ : R + → R + with ζ (0) = 0 .
We then apply Lemma 5 to lower bound the LHS and upper bound the RHS:
[Equation]
from which we have:
[Equation]
Setting Ψ( z ) =
(1 + c max ) · ζ (cid:16) 1 c min · z (cid:17) completes the proof.
We are now ready to prove Theorem
[Equation]
Proof of Theorem 4.
We have:
[Equation]
We ﬁrst bound the second term below:
[Equation]
where the last step uses the fact
that 1 ( r ∗ ( x ) < 0) ≤ 1 and by deﬁnition of h ∗ , E y | x [ 1 ( y (cid:54) = h ( x ))] ≥ E y | x [ 1 ( y (cid:54) =
h ∗ ( x ))] , for all classiﬁers h .
Substituting this back into (21) to bound the second term, and applying Lemma 6 to bound the ﬁrst term, we have:
[Equation]
x [Equation] where Ψ is some increasing function with Ψ(0) = 0 , and in the second inequality, we use the fact that h ∗ is the Bayes-optimal classiﬁer for the misclassiﬁcation error.
B Learning to defer as cost-sensitive learning
For a classiﬁer h : X → Y , the misclassiﬁcation error R ( h ) =
P ( y (cid:54) =
h ( x )) assumes that all mispredictions are equally undesirable, and that the classiﬁer can only output labels from Y .
More generally, cost-sensitive classiﬁcation [11] seeks h : X → ˆ Y that minimises
[Equation]
for cost function α : X × Y × ˆ Y → R + and prediction space ˆ Y .
When ˆ Y =
Y and α ( x, y, ˆ y )
= 1 ( y (cid:54) =
ˆ y ) ,
R cs ( h ) =
R ( h ) .
The Bayes-optimal cost-sensitive classiﬁer is
[Equation]
In the learning to defer ( L2D ) problem [18], one seeks a classiﬁer that can either make a standard prediction in Y , or defer its prediction to an expert model h exp : X → Y .
Invoking an expert carries an associated sample-dependent cost c exp ( x, y ) > 0 , so as to prevent the classiﬁer from deferring on all samples.
Concretely, let ˆ Y =
Y ∪ {⊥} , where ⊥ denotes the “defer” option.
Consider a classiﬁer ¯ h : X → ˆ Y equipped with a “defer” option ⊥ .
Our goal is then to minimise
[Equation]
for α ( x
[Equation]
y
[Equation]
ˆ
[Equation]
if
[Equation]
(cid:54)
[Equation]
c
[Equation] else. [Equation]
C On the Bayes-optimal deferral logit and probability
We now detail the form of the Bayes-optimal defer logit ¯ f ∗⊥ ( x ) for the cost-sensitive softmax crossentropy, assuming that all other logits ¯ f 1 ( x ) , . . .
, ¯ f L ( x ) are ﬁxed, and chosen arbitrarily.
This is slightly more general than (5), which considered the joint Bayes-optimal solution for all logits.
This analysis highlights two points: ﬁrst, even when the other logits ¯ f 1 ( x ) , . .
, ¯ f L ( x ) are chosen arbitrarily, there is no dependence of the optimal defer probability ¯ p ∗⊥ ( x ) on the logits ¯ f 1 ( x ) , . . .
, ¯ f L ( x ) .
Second, as a qualiﬁer to the above, there is however a dependence of the optimal defer logit ¯ f ∗⊥ ( x ) on the logits ¯ f 1 ( x ) , . . .
, ¯ f L ( x ) .
From (4) [Equation] the cost-sensitive softmax cross-entropy has conditional risk
L ( x, ¯ f
[Equation]
x
[Equation]
)) ¯ Z ( x ) , where ¯ Z ( x ) .
= Z ( x ) + exp( ¯ f ⊥ ( x )) and Z ( x ) .
= (cid:80) y (cid:48) ∈ Y exp( ¯ f y (cid:48) ( x )) . Thus, the terms involving ¯ f ⊥ ( x ) are
[Equation]
At optimality, for any choice of ¯ f 1 , . . .
, ¯ f L , we thus have
[Equation]
Thus, for any ¯ f 1 , . . .
, ¯ f L , the probability assigned to the deferral label will be independent of these logits.
Note however that the deferral logit ¯ f ∗⊥ itself will necessarily depend on these logits: intuitively, it serves to re-normalise the softmax probabilities of these logits to ensure ¯ p ⊥ takes the desired form.
Indeed, some simple algebra reveals that
[Equation]
Thus, the Bayes-optimal defer logit has a non-linear dependence on the logits ¯ f 1 , . .
, ¯ f L .
We further remark that when the model class is ¯ f is of insuﬃciently high capacity, it may not be feasible to achieve this Bayes-optimal solution (even though ¯ p ∗⊥ ( x ) is relatively simple).
For example, suppose we parameterise ¯ f y (cid:48)
( x ) = w (cid:62) y (cid:48)
Φ( x ) + b y (cid:48) .
Then, it is not possible to express ¯ f ∗⊥ ( x ) from (24) in the form w (cid:62)⊥ Φ( x ) + b ⊥ , since log (cid:104)(cid:80) y (cid:48)
∈ Y exp( ¯ f y (cid:48) ( x ))
(cid:105) is non-linear in Φ( x ) .
As a ﬁnal remark, we note that the picture changes when considering the one-versus-all loss: here, regardless of the choice of ¯ f 1 , . . .
, ¯ f L the Bayes-optimal solution ¯ f ∗⊥ will be ψ −
[Equation]
(cid:19) .
This is owing to the decoupled nature of the loss; consequently, there is no inﬂuence of the other logits on ¯ f ∗⊥ .
D General one-versus-all loss
We now present a more general form of the one-versus-all loss, where we allow for a generic cost function α ( x, y, ¯ h ( x )) following (23).
Consider a loss of the form
[Equation]
(25) for a yy (cid:48) ( x ) , b yy (cid:48)
( x ) ∈ R .
The conditional risk is
[Equation]
where A y (cid:48) ( x ) .
=
E y | x [ a yy (cid:48)
( x )] and B y (cid:48) ( x ) .
=
E y | x [ b yy (cid:48) ( x )] .
The Bayes-optimal scorer for a strictly proper composite loss with inverse link ψ is thus
[Equation]
Now observe that the Bayes-optimal solution for the cost-sensitive problem (22) is expressible as
[Equation]
where α max is an upper bound on α ( x, y, y (cid:48) ) .
Thus, if we set
[Equation]
the Bayes-optimal scorer in (26) will precisely yield ψ ( f ∗ y (cid:48) ( x ))
= 1 −
E y | x [ α ( x,y,y (cid:48) )
α
[Equation]
As a sanity check, when α corresponds to the learning to defer risk, we have
[Equation]
which is seen to yield the OvA loss in (10).
E Summary of losses for learning to defer
Table 1 summarises various approaches for learning to defer.
Each method employs a base loss (cid:96) ( y, x, f ( x ); c base ) to train the base model with a base deferral cost c base , and optionally optimizes a rejector model using the rejector loss provided in (14) with the user-speciﬁed ﬁxed deferral cost c 0 : (cid:96) rej ( x, y, r ( x ); c 0 ) =
c mod ( x, y ) · φ ( r ( x ))
+ ( c 0 +
( y (cid:54) =
h exp ( x ))) · φ ( − r ( x )) .
Prior methods by Mozannar and Sontag [19] and Verma and Nalisnick [34] include the user-speciﬁed cost c 0 in the base loss, i.e., set c base =
c 0 , and do not train an explicit rejector.
The proposed methods in this paper set c base = 0 in the base loss, and include c 0 either as a part of a post-hoc thresholding step, or as a part of the rejector loss in an explicit post-hoc training step.
We also include a variant where one minimizes the standard softmax cross-entropy loss (cid:96) CE
( y, f ( x ))
=
log (cid:2) (cid:80) y (cid:48)
∈
Y e f y (cid:48)
( x ) (cid:3) − f y ( x ) to train the base model, and trains the rejector in a post-hoc manner.
F Parameterising the post-hoc rejector
In constructing the rejector model r : X → R , it may be beneﬁcial to paramterize it in terms of the base model’s probabilities.
For example, if the base loss is the CSS loss in (8), then we may want to use a rejector of the form:
[Equation]
where ¯ p denotes a softmax transformation of the L + 1 logits from the ﬁxed base model ¯ f , and g ⊥ :
X → R is a example-dependent bias term that we explicitly train.
Notice that if the ﬁrst L probabilities from the base model accurately estimate the underlying conditional-class probabilities, i.e., ¯ p y
( x ) ∝ P ( y | x ) , ∀ y ∈
[ L ] , and the reject probability is an accurate estimate of the expert’s error, i.e., ¯
p ⊥ ( x ) ∝ P ( y (cid:54) = h exp ( x )) , then r ( x ) > 0 matches the
Bayes-optimal deferral decision when g ⊥ ( x ) =
c 0 .
In practice, the base model may not accurately estimate either P ( y | x ) or the expert’s error, and therefore the additional term g ⊥ ( x ) that we train can be seen as correcting errors in the base model’s estimates.
Table 1 also presents the rejector parameterisation for other base losses, such as the OvA loss and the
SOvA loss.
In each case, the form of the rejector is chosen to recover the Bayes-optimal decision rule when the base model provides calibrated probability estimates for P ( y | x ) and P ( y (cid:54) =
h exp ( x )) .
We employ these parameterisations in all our experiments in §5. [Table] ) =
g ⊥ ( x ) − max y ∈
[ L ] p y ( x ) (cid:96) CSS ( y,x, ¯ f ( x );0) (cid:96) rej ( y,x,r ( x ); c 0 ) , where r ( x )
[Table]
r ( x ) = g ⊥ ( x ) + ¯ p ⊥ ( x ) [Table] −
[Table]
p
[Table]
( x )
[Table]
y ∈
[ L ]
[Table]
p y ( x )
[Table]
p
[Table]
x ) (cid:96) OVA ( y,x, ¯ f ( x );0) (cid:96) rej ( y,x,r ( x )
[Table]
) , where
[Table]
[Table]
[Table]
[Table]
x )
[Table]
) , where
[Table]
r
[Table]
G Relation to diﬀerentiable triage paradigm
Okati et al.
[22] consider a variant of the L2D setup where one is given a budget on the fraction of examples on which the model is allowed to defer.
The goal is to now learn a base model and deferral rule satisfying this budget constraint.
An analogous constrained setting has also been considered in the adaptive inference literature [15].
One potential approach to incorporating post-hoc techniques into this framework is to formulate an equivalent Lagrangian saddle-point problem, and perform alternating stochastic updates on ¯ f , and updates on the multiplier associated with the budget constraint.
Further exploring this would be of interest.
H Details of experiment parameters H.1 Synthetic dataset
We construct a distribution P ( x, y )
= (cid:80) a ∈{ 0 , 1 } P ( x, y, a ) as follows.
We ﬁx P ( a = 1) = 0 . 5 ,
P ( y = 1 | a ) = 0 .
For subgroup a = 0 , we set P ( x |
y, a = 0)
=
N ( µ ( y,a ) , Σ ( y,a ) ) .
We draw the elements of µ (0 , 0)∼ Unif( − 2 , − 1) , µ (0 , 0)∼ Unif( − 1 , − 1) , and set µ (1 , 0)= µ (0 , 0)+ z for z ∼ Unif(3 , 4) .
We set Σ (0 , 0) =
Σ (1 , 0) to be the identity matrix scaled by a random constant s ∼ Unif(0 , 1) .
Observe that the Gaussians for a = 0 both have identical isotropic covariance.
The
Bayes-optimal solution for the subgroup P ( x, y | a = 0) is a linear classiﬁer over the raw features x .
For subgroup a = 1 , we draw θ ∼ Unif(0 , 2 π ) , and set r = 5 + y .
We then collect samples of the form ( r cos θ, r sin θ ) .
H.2 Real-world datasets
For all datasets, we perform minibatch SGD optimisation with momentum 0 .
Dataset speciﬁc settings are given below.
Quantity
CIFAR-
ImageNet
Weight decay
[Figure]
Epochs
Batch
[Figure]
Learning
[Figure]
Learning rate: annealing
Decay by 0 . 1 after 96th, 192nd, 224th epoch
Cosine
Data augmentation [14] [12]
[Figure]
I Additional experiments: underﬁtting of existing losses I.1 Underﬁtting of cost-sensitive softmax cross-entropy
For the setting in Figure 1, Figure 7 shows plots of the logit margins f y ( x ) − max y (cid:48) (cid:54) =
y f y (cid:48)
( x ) , and the probability margins p
y ( x ) − max y (cid:48)
(cid:54) =
y p y (cid:48)
( x ) .
Both of these degrade as c 0 increases, indicating that the probability of the true label is increasingly confusable with that of other labels.
I.2 Underﬁtting of one-versus-
all loss
For the setting in Figure 1, Figure 7 shows analogous plots for the one-versus-all loss of Verma and
Nalisnick [34].
From the top left panel, we see that as c 0 increases, the training accuracy degrades.
There is a sharper transition here in the margin distributions as c 0 becomes non-zero.
Nonetheless, we again see that while c 0 = 0 produces highly conﬁdent predictions on the training set (e.g., from the entropy plot), the introduction of c 0 > 0 makes the predictions much less conﬁdent (e.g., as evident from the entropy and margin plots).
I.3 Impact of expert cost function on underﬁtting
We emphasise here that the underﬁtting issues above are a consequence of incorporating a ﬁxed deferral cost c 0 > 0 in addition to the expert error probability.
In settings where c 0 = 0 , or where the expert is assumed to be perfect, such an issue is not present.
The former has been considered as part of the plots in the previous section.
The latter is potentially applicable in some settings, and indeed corresponds to the classic learning to reject setting [2, 9, 6].
Here, we may set c max = 1 , and the cost-sensitive softmax cross-entropy simpliﬁes to
[Equation]
which is similar in nature to (8).
As with (8), we will not exhibit severe underﬁtting as a result of the loss only considering ¯ p y and ¯ p ⊥ , as opposed to all possible labels. [Figure] [Figure]
To conﬁrm this [Figure] we repeat the CIFAR-100 experiment considered in Figure 1 with a constant cost of c exp ( x, y )
=
c 0 .
I.4 Impact of number of labels on underﬁtting
Figure 10 shows an analogue of Figure 1 for the CIFAR-10 dataset.
We see largely similar trends, although the magnitude of degradation is not as severe as CIFAR-100.
This is consistent with our [Figure] analysis of the large L setting being problematic, since the amount of label smoothing on all labels scales with L .
J Additional experiments: quality of one-versus-all probability estimates
We demonstrate that the probabilities ¯ p 1 , . . .
, ¯ p L for the standard labels obtained from the one-versusall loss ( OvA ) may underperform those obtained from the hybrid softmax cross-entropy plus OvA loss.
Figure 11 compares the distribution of the log-loss − log ¯ p y ( x ) , the entropy normalised by log L , and the logit and probability margins.
We see that the hybrid softmax cross-entropy plus OvA loss results in consistently lower log-loss values, as well as entropies and margins.
Intuitively, the latter suggests that in the regular OvA loss, there are multiple labels y (cid:48) for which ¯ f y (cid:48) is large, resulting in multiple large ¯ p y (cid:48) values.
As a qualifying comment, the value of OvA probabilities has been demonstrated in out-of-domain settings [23]; it is of interest to further study the intersection of this with the deferral setup considered in the present work.
[Figure]
We see that the hybrid softmax cross-entropy plus OvA loss results in consistently lower log-loss values, as well as entropies and margins.
K Additional experiments: OvA and SOvA losses under diﬀerent post-hoc schemes
We explained in §4.1 why the hybrid softmax cross-entropy plus OvA loss is well-suited for use with post-hoc thresholding approach, and provided empirical evidence in the previous section to showcase why it is a better alternative to the standard OvA loss.
For the sake of completeness, we compare the performance of these losses when combined with both the post-hoc schemes proposed in this paper.
Figure 12 presents results for all four combinations ( OvA and SOvA with c 0 = 0 coupled with either post-hoc thresholding and post-hoc training) on the CIFAR-10 and CIFAR-100 datasets in a learning to defer setting where the base models can defer to “specialist” experts (§5.2).
On both datasets, it is one of the SOvA -based post-hoc approaches that performs the best for most operating points.
Between the two post-hoc thresholding approaches, the one that uses the SOvA loss is clearly seen to yield signiﬁcantly higher accuracies than the OvA loss.
We also plot the results from two additional baselines.
The ﬁrst is post-hoc training on a base model trained with the standard softmax cross-entropy ( CE ) loss.
The fact that this baseline is seen to under-perform for most deferral costs highlights the beneﬁt of using a loss that explicitly allows the base model to defer to the expert.
The second is the conﬁdence-based approach of Raghu et al.
[26], where one trains a base model using the CE loss, then trains a model to get conﬁdence estimates for the expert, and compares the base model’s conﬁdence scores with those estimated for the expert to decide whether to defer on an example.
For the conﬁdence estimation, we adopt the same architecture as the post-hoc rejector we use in our post-hoc training scheme, namely a linear model on top of the embeddings learned by the ﬁrst model, and train it with the logistic loss to predict whether the expert classiﬁes a given training example correctly.
As seen, at least one of the proposed post-hoc schemes outperforms the baseline on almost all operating points, highlighting the beneﬁt of training the defer logit jointly with the base model.
[Figure]
L Additional experiments: OvA and CSS losses with diﬀerent post-hoc thresholding rules
In another set of experiments we train the base model using the OvA and CSS losses (with c 0 = 0 ), and apply the post-hoc thresholding rule in (11) to defer on an example.
We also include a variant of this approach, where we train using the OvA and CSS losses (with c 0 = 0 ), ignore the defer logit, and apply
Chow’s rule [8] with c exp ( x, y ) =
c 0 to defer on an example.
The latter approach does not explicitly take the expert’s error into account.
Figure 13 presents comparisons of all four combinations OvA and
CSS losses with two the diﬀerent thresholding schemes.
We additionally include the standard Chow’s rule for learning to reject with a constant rejection cost c 0 ( CE + Chow), and the proposed scheme of training the base model using the SOvA loss (with c 0 = 0 ) and applying our post-hoc thresholding rule ( SOvA + Post-hoc Thresholding).
The proposed post-hoc thresholding approach is seen to yield a higher accuracy than all the baselines at most operating points.
Also, we see that, as expected, the use
[Figure]
of Chow’s thresholding rule generally under-performs a post-hoc thresholding scheme that explicitly takes the expert’s error into account.
M Additional plots: deferral risk in (1) vs c
The plots in the main body show the tradeoﬀ between the fraction of samples deferred to the experts, and the accuracy.
In these plots, the role of the ﬁxed cost c 0 is implicit , with high ﬁxed costs translating to a small fraction of samples deferred to the expert.
We may also make the role of c 0 explicit , by studying how varying c 0 aﬀects the deferral risk in (1).
These are presented in Figures 14 and 15.
As with Figures 5 and 6, we see that the CSS and OvA losses tend to underperform at high ﬁxed costs c 0 .
Figures 16 and 17 further show the accuracy of the ﬁnal classiﬁer as a function of c 0 .
Again, we observe that the CSS and OvA accuracy tends to suﬀer as c 0 is increased.
With the adaptive inference experiments, CSS and OvA tend to defer a large proportion of the examples to the expert even at higher costs (Figure 18)
[Figure]
and as a result yield a higher accuracy, but with poorer deferral risk
[Figure] [Figure]
[Figure] [Figure]
[Figure]
S a m p l e s d e f e rr e d t
o
e x p e r t ( % )
[Figure]
d
[Figure]
f
[Figure]
rr
[Figure]
d t o e x p e r t
[Figure]