Post-hoc estimators for learning to defer to an expert
Anonymous Author(s)
AﬃliationAddress email
Abstract
Many practical settings allow a classiﬁer to defer predictions to one or more costlyexperts .
For example, the learning to defer paradigm allows a classiﬁer to defer toa human expert, at some monetary cost.
Similarly, the adaptive inference paradigmallows a base model to defer to one or more large models, at some computationalcost.
The goal in these settings is to learn classiﬁcation and deferral mechanisms tooptimise a suitable accuracy-cost tradeoﬀ.
To achieve this, a central issue studiedin prior work is the design of a coherent loss function for both mechanisms.
In thiswork, we demonstrate that existing losses can underﬁt the training set when there isa non-trivial deferral cost, owing to an implicit application of a high level of labelsmoothing.
To resolve this, we propose two post-hoc estimators that ﬁt a deferralfunction on top of a base model, either by threshold correction, or by learningwhen the base model’s error rate exceeds the cost of deferring to the expert.
Bothapproaches are equipped with theoretical guarantees, and empirically yield eﬀectiveaccuracy-cost tradeoﬀs on learning to defer and adaptive inference benchmarks.
Introduction: learning to defer to an expert model
Supervised classiﬁcation conventionally considers learning a single model with good average-casetest performance.
However, many practical settings allow for choosing amongst multiple models toclassify a given sample.
For example, in the learning to defer ( L2D ) paradigm [17], a base classiﬁermay be used in conjunction with a human expert .
Speciﬁcally, the classiﬁer has the option of deferringto the expert at the expense of incurring some additional cost (e g., monetary expense).
The goal isto learn a classiﬁer and deferral function achieving a suitable accuracy-cost tradeoﬀ. This problemhas received a surge of interest [25, 18, 21, 33] given the increasing use of machine learning inhigh-stakes decision making.
Interestingly, similar considerations are also found in the problem ofadaptive inference for resource-constrained prediction [12]: here, a base model can choose to defer toone or more experts, which are themselves learning models with higher capacity and inference cost.
Similar to L2D, the goal is to achieve a suitable tradeoﬀ between accuracy and (inference) cost.
A key problem in such settings is deciding when the base model should defer its prediction to theexpert(s).
An intuitive strategy is to defer when the model has low prediction conﬁdence (e.g.,classiﬁcation margin).
While theoretically grounded [7], this strategy can under-perform when thebase model has low capacity [8].
An alternate strategy is to explicitly formulate and optimise a jointloss function for the base classiﬁer and deferral function.
This has been followed with success in the
L2D literature, particularly with the cost-sensitive softmax cross-entropy loss ( CSS ) of Mozannar and
Sontag [18], and one-versus-all ( OvA ) loss of Verma and Nalisnick [33].
While the above losses have proven eﬀective, they are not without limitations.
Indeed, both focus onthe setting where the cost of deferring is only given by the probability that the expert makes a mistakeon a sample; this does not consider the additional ﬁxed cost of querying an expert in the ﬁrst place.
As we shall see (§3), incorporating any non-zero ﬁxed cost makes these losses prone to underﬁtting ;cf.
Figure 1.
This behaviour is owing to the fact that the losses implicitly apply a high level of label
[Figure] smoothing . The setting of non-zero ﬁxed cost is particularly important in adaptive inference settings,where querying an expert inherently involves paying an increased computational cost.
In this work we resolve this issue via deferral schemes based on post-hoc estimators .
In a nutshell,we follow a two-step procedure wherein we ﬁt a deferral function on top of a base model, either bythreshold correction, or by learning to predict when the base model’s error rate exceeds the deferralcost.
This allows us to build on top of the standard (zero ﬁxed cost) CSS and OvA solutions, resultingin an empirically eﬀective strategy with theoretical guarantees.
In sum, our contributions are:
(i) we identify limitations in prior losses for L2D, particularly when the cost of deferring to an expert involves a non-zero ﬁxed cost in addition to the probability of the expert making a mistake (§3)(ii) we propose new post-hoc estimators for learning a deferral function, which overcome the abovelimitation while enjoying theoretical guarantees (§4, Proposition 2, Lemma 3)(iii) we empirically show that the new post-hoc estimators work well on synthetic and real-worldbenchmark datasets for both L2D and adaptive inference settings (§5).
In the course of our analysis, we additionally generalise the OvA loss to reliably account for generaldeferral costs ((10), Lemma 1), and to reliably estimate label probabilities in practical settings (12).
Background and notation
Our interest is in learning settings where a classiﬁer is equipped with the option of deferring to anexpert model.
For concreteness, we shall focus our exposition on the learning to defer paradigm.
2.1 Multi-class classiﬁcation
Fix an instance space X and label space Y =
[ L ] .
= { 1 , 2 , . . .
, L } , and let P be a distribution over 58 X × Y .
Given a training sample S =
{ ( x n , y n ) } n ∈ [ N ] drawn from P , multi-class classiﬁcationseeks a classiﬁer h : X → Y with low misclassiﬁcation error R ( h )
=
P ( y 6 = h ( x )) .
We mayparameterise h as h ( x ) =
argmax y
∈ Y f
y 0 ( x ) for a scorer f : X → R L .
Such a scorer may be learnedby minimising the empirical surrogate risk
ˆ R ( f )
= 1 N
P n ∈ [ N ] ‘ ( x n , y n , f ( x n )) for some loss‘ : X × Y × R L → R + , such as the softmax cross-entropy ‘ ( x, y, f ( x ))
= log hP y
∈ Y e f y
( x ) i − f y ( x ) .
For binary classiﬁcation problems with L = 2 , it is common to enforce f 2 ( x ) = − f 1 ( x ) and set‘ ( x, y, f ( x ))
=
φ ( f y ( x )) for some margin loss φ : R → R + , such as φ ( z )
=
log(1 + e − z ) .
2.2 Learning to defer (L2D) problem
In the learning to defer ( L2D ) problem [17], one seeks a classiﬁer that can either make a standardprediction in Y via some base model, or defer its prediction to an expert model h exp : X → Y , at the expense of incurring a sample-dependent cost c exp ( x, y ) > 0 .
Typically, c exp takes into accountboth the ﬁxed cost of deferring to the expert, which we will assume to be a constant c 0 ∈ [0 , 1] (e.g.,normalised monetary cost), and an estimate of the probability that the expert makes a mistake on theexample.
Canonical examples include c (1) exp ( x, y ) =
c 0 and c (2) exp ( x, y )
=
c 0 +
y 6 = h exp ( x )) .
Thechoice c (1) exp ( x, y ) corresponds to learning with a reject option or abstention [7, 2, 8, 5].
Given a suitable c exp ( x, y ) , the learning problem may be formalised as follows.
Consider a classiﬁer¯ h : X → Y ∪ {⊥} equipped with a “defer” option ⊥ .
Our goal is to minimise
[Equation]
Intuitively, when ¯ h chooses not to defer, we incur the usual misclassiﬁcation error; otherwise, we incurthe cost c exp ( x, y ) of invoking the expert h exp .
One can cast (1) as an instantiation of cost-sensitivelearning [10]; see Appendix B. We remark that Okati et al.
[21] considered a slightly diﬀerent setting,wherein one enforces a hard constraint on the expert cost c exp ( x, y ) .
We focus on a soft additivepenalty in (1), but we may handle hard constraints via standard Lagrangian theory; see Appendix G.
In practice, it is useful to parameterise ¯ h in terms of a scorer ¯ f : X → R L +1 with label logits{ ¯ f 1 ( x ) , . . .
, ¯ f L ( x ) } , and “defer” logit
¯ f ⊥ ( x ) .
We then predict via
[Equation]
In practice, we may parameterise ¯ f via neural network logits: for any y 0 ∈ Y ∪{⊥} , ¯ f y 0 ( x ) =
w > y
Φ( x ) 83 for weights w y 0 and embedding Φ .
This parameterisation induces implicit sharing amongst all logits.
One conceptually useful quantity is the Bayes-optimal ¯ h ∗ =
argmin ¯ h : X → Y ∪{⊥} R def (¯ h ) , which is
[Equation]
Intuitively, we defer iﬀ the expected cost of querying the expert is smaller than the expected error ofthe base model.
For the constant cost function c (1) exp ( x, y ) , this is known as Chow’s rule [7].
2.3 Losses for learning to defer
We review some popular recent strategies to learn ¯ f (and thus ¯ h ) below.
Conﬁdence thresholding .
For c exp ( x, y )
=
c 0 , one strategy is to learn ¯ f 1 , . .
, ¯ f L by minimising thesoftmax cross-entropy, and ﬁx ¯ f ⊥ ( x ) = 1 − c 0 .
This conﬁdence thresholding approach [20] mimicsthe Bayes-optimal classiﬁer (3).
A related series of approaches [26, 37, 1] make deferral decisions bycomparing the base model conﬁdence with the conﬁdence in the expert being correct.
When the basemodel has low-capacity, such approaches may be sub-optimal as the base model may sub-optimallyallocate its ﬁnite capacity by classifying examples that the expert can easily predict.
Cost-sensitive softmax cross-entropy ( CSS ) .
To overcome the limitations of conﬁdence thresholding,it is desirable to explicitly learn ¯ f ⊥ as well.
To this end, Mozannar and Sontag [18] proposed acost-sensitive softmax cross-entropy ( CSS ): for constant upper bound c max ≥ c ( x, y ) , this is
[Equation]
where ¯ p y ( x ) = exp( ¯ f y ( x )) / P
[Equation] ∈ Y [Equation] {⊥} exp( ¯ f y 0 ( x )) is the softmax distribution over the standardand the defer label.
For c exp ( x, y )
=
c 0 +
y 6 = h exp ( x )) , we have c max = 1 + c 0 .
The loss ‘ CSS is
Fisher-consistent [18, Proposition 1], with Bayes-optimal prediction
[Equation]
One-versus-all ( OvA ) .
Despite the Fisher consistency of ‘ CSS , Verma and Nalisnick [33] observed thatempirically, the loss may not produce calibrated estimates of the expert’s error probability.
Verma and
Nalisnick thus proposed a one-versus-all ( OvA ) loss for the cost function c exp ( x, y )
=
1 ( y 6 = h exp ( x )) :
[Equation]
f ⊥ ( x )) + c exp ( x, y ) · φ ( − ¯ f ⊥ ( x )) .
Here, φ : R → R + is a binary proper composite loss [27], e.g., φ ( z )
=
log(1 + e − z ) .
Such losseshave an associated inverse link function ψ : R → [0 , 1] , and produce Bayes-optimal scorer
[Equation]
Intuitively, both (4) and (6) seek to have the ﬁrst L logits { ¯ f 1 ( x ) , . .
, ¯ f L ( x ) } estimate P ( y | x ) , andthe reject logit ¯ f ⊥ ( x ) model the cost of querying the expert.
With this, the highest scoring logitprovides us with an estimate of the Bayes-optimal deferral decision (3).
Limitations of existing learning to defer losses
We now study existing losses for learning to defer more carefully, and show that they may underﬁt inan important practical setting.
Speciﬁcally, recall that a canonical choice of deferral cost functionin (1) is c exp ( x, y )
=
1 ( y 6 = h exp ( x ))
+ c 0 , for constant c 0 ≥ 0 .
Here, c 0 represents a ﬁxed cost ofquerying the expert, which is independent of whether the expert misclassiﬁes the sample.
Thesetting of non-zeroﬁxedcost c 0 > 0 isof practicalinterest; e.g., thisisintrinsictoadaptiveinferencesettings, wherein querying the expert model fundamentally involves increasing a computationaloverhead (see §4.3 for more discussion).
Thus, an ideal surrogate loss for (1) should be performantfor c 0 > 0 .
Unfortunately, we now show that this does not hold for the CSS and OvA losses.
3.1 Underﬁtting of cost-sensitive softmax cross-entropy when c 0 >
The cost-sensitive softmax cross-entropy ( CSS ) is in principle applicable for any deferral cost c exp ( x, y ) ,and in particular c exp ( x, y )
=
c 0 +
y 6 = h exp ( x )) for any c 0 ≥ 0 .
However, the focus in Mozannarand Sontag [18] was the case c 0 = 0 .
In fact, we now show that when c 0 > 0 , the loss can underﬁt .
Tosee this, note that the tightest possible maximal cost is c max = 1 + c 0 .
With this choice, (4) becomes
[Equation]
When c 0 = 0 , the second term vanishes, and we have a conic combination of the log-loss on thetrue and deferral labels.
When c 0 > 0 , the second term is active, and applies a form of labelsmoothing [30]: we allow for all labels y
∈ Y − { y } to be potential “positives” for x , with a weight 127 of c 0 ∈ [0 , 1] .
Put diﬀerently, we treat the ground-truth label distribution as a mixture of a one-hotdistribution on y , and a uniform distribution over all labels.
As c 0 → 1 − , we apply a large amount ofsmoothing on labels other than y ; this can be problematic.
Concretely, consider an x where the expertgives the wrong prediction, i.e., P ( y = h exp ( x ))
= 0 .
Then, the Bayes-optimal solution (5) will be
[Equation]
Observe that for c 0 > 0 , as L increases, the range of values in ¯ p ∗ shrinks: indeed, the gap between theoptimal probability for true label y and a competing label y 0 will be O (1 /L ) .
This small probabilitymargin makes learning challenging: given a ﬁnite sample, it will be easy for the model to mistakenlyswap the true label y with some competing label y 0 .
Thus, in settings where L is moderately large,and where there is a non-zero ﬁxed deferral cost c 0 > 0 , ‘ CSS may underperform.
Such issues havealso been observed [6] for a similar loss developed for label noise [24].
As an illustration, we consider in Figure 1 a learning to defer setting comprising a ResNet-8 base modeland a ResNet-32 “expert” on CIFAR-100.
We assume a deferral cost c exp ( x, y ) =
c 0 +
y 6 = h exp ( x )) ,where c 0 is varied in [0 , 1] .
For each c 0 , we train the base model using (4), and report the train andtest accuracy.
As c 0 increases, the base model shows signiﬁcant training accuracy degradation, i.e., itunderﬁts.
The high level of label smoothing plays a role: in the right panel, we see that the entropy ofthe model’s predicted distribution over labels increases with c 0 .
(The entropy is normalised by log Lto lie in [0 , 1] .)
See Appendix I for additional plots, and Appendix I.3 for results when c exp ( x, y )
=
c 0 .
3.2 Underﬁtting of one-versus-all loss when c 0 > 0
The one-versus-all ( OvA ) loss of Verma and Nalisnick [33] was presented for the speciﬁc case ofc 0 = 0 ; as stated, the loss is not applicable when c 0 > 0 .
We may however extend this loss to a general c exp ( x, y ) with upper bound c max ≥ 1 (see Appendix D for a more general derivation):
[Equation]
Compared to (6), we have an additional weighted sum over each φ y 0 ( x ) .
As with OvA , this loss resultsin a coherent Bayes-optimal solution when φ is a strictly proper composite loss [4, 27].
Recall thatsuch losses are characterised by an inverse link function ψ : R → [0 , 1] , such as the logistic lossφ ( z )
=
log(1 + e − z ) with sigmoid inverse link ψ ( v ) =
(1 + exp( − v )) − 1 .
Lemma 1.
Suppose φ is a strictly proper composite loss with inverse link function ψ .
Then, theone-versus-all loss in (10) is calibrated for the learning to defer risk (1) , with Bayes-optimal scorer
[Equation]
Unlike the optimal scorer for the CSS loss, here
P
[Equation]
[Equation]
] ψ ( ¯
[Equation]
y [Equation] x )) 6 = 1 .
Nonetheless, the highestscoring logit agrees with the Bayes-optimal classiﬁer (3) for learning to defer.
Reviewing (10), we see that when c max > 1 , we apply a form of label smoothing by additional termsof the form
[Equation]
¯ f y 0 ( x )) , i.e., we eﬀectively treat all y 0
= y as potential positives.
As with thecost-sensitive softmax entropy, this can result in underﬁtting; recall that with c exp ( x, y )
=
c 0 +
y 6 =h exp ( x )) , we have an upper bound c max = 1 + c 0 > 1 .
We empirically conﬁrm in Appendix I.2 thatwe indeed observe analogous underﬁtting behaviour as per the CSS loss in Figure 1.
The above shows that the CSS and OvA losses have a subtle limitation when c 0 > 0 .
This is not inconﬂict with the Fisher-consistency of these losses: the latter establishes that the asymptotic minimiserof these losses agrees with the Bayes-optimal classiﬁer (3).
Our analysis explicates that the preciseform of these Bayes-optimal solutions may not be amenable to learning from a ﬁnite sample .
Since both losses seamlessly handle c 0 = 0 , a natural question is whether one can leverage thissolution to help guide the one for c 0 > 0 .
We now present post-approaches that do precisely this.
Post-hoc approaches to learning to defer
We now present two simple post-hoc strategies for learning to defer, which build on the CSS and OvAsolutions for c 0 = 0 via threshold correction and re-training the deferral function.
4.1 Post-hoc threshold correction
Our ﬁrst post-hoc strategy (Figure 2) exploits the form of the Bayes-optimal classiﬁer in (3): to mimicthis classiﬁer for the cost function c exp ( x, y )
=
1 ( y 6 = h exp ( x ))
+ c 0 , it suﬃces to obtain estimatesof P ( y | x ) and the expected deferral cost E [ c exp ( x, y )]
=
P ( y 6 = h exp ( x ))
+ c 0 .
Consequently,suppose we learn models π : X → [0 , 1] L and (cid:15) :
X → [0 , 1] estimating P ( y | x ) and P [Equation] y 6 = h exp ( x ))respectively.
Then, for any c 0 ≥ 0 , we can approximate (3) via the classiﬁer
[Equation]
Intuitively, this procedure ought to perform well when π ( x ) is close to η ( x ) .
=
[ P ( y | x )] y ∈ Y , and(cid:15) ( x ) is close to P ( y 6 = h exp ( x )) .
The following excess risk bound for ¯ h formalises this intuition.
Proposition 2.
Pick any π : X → [0 , 1] L and (cid:15) :
X → [0 , 1] .
Let ¯ h be the corresponding classiﬁerin (11) .
Then, the excess risk of ¯ h compared to the Bayes-optimal classiﬁer ¯ h ∗ (3) is:
[Equation]
Crucially, observe that we can learn π and (cid:15) once , and then sweep across diﬀerent c 0 values in (11).
Thus, we can attempt to leverage the solutions for either the CSS or OvA losses with c 0 = 0 — recallingthat these do not exhibit underﬁtting, unlike when c 0 > 0 — and then simply vary c 0 post-hoc.
Indeed,
Trainingsample
[Figure]
SOvA
[Figure] [Figure]
c
[Figure]
π y ( x ) =
exp(¯ f y
x ))
P y 0 ∈
Y exp(¯ f y
( x )) (cid:15) ( x ) =
ψ (¯ f ⊥ ( x ))
Compute statistics
Classify via (11) for any c
Final classiﬁer { ( x n ,y n )
[Figure]
Nn
[Figure]
Logits ¯ f 1 ,..., ¯ f L
Deferlogit ¯ f ⊥ π ( x ) ,(cid:15) ( x ) both losses’ Bayes-optimal solutions (5), (7) for c 0 = 0 suggest that a transform of ¯ f 1 , . . .
, ¯ f L provideestimates of P ( y | x ) , while a transform of ¯ f ⊥ provides an estimate of P ( y 6 = h exp ( x )) .
Unfortunately, in practice, both losses are potentially suboptimal to learn π, (cid:15) .
For the CSS loss, Vermaand Nalisnick [33] showed it does not provide calibrated estimates of P ( y 6 = h exp ( x )) .
On the otherhand, while the OvA loss provides calibrated estimates of P ( y 6 = h exp ( x )) , its probability estimatesof P ( y | x ) may underperform when L is large (see Appendix J).
We overcome this with a simplesolution: we estimate π via the softmax cross-entropy, and (cid:15) via the OvA loss.
This is achieved via ahybrid softmax cross-entropy plus OvA loss ( SOvA ):
[Equation]
where p y ( x ) = exp( ¯ f y ( x )) / P
[Equation]
∈ Y exp( ¯ f y 0 ( x )) .
Following the Bayes-optimal solution, we setπ ( x ) = p ( x ) and (cid:15) ( x ) = ψ ( ¯ f ⊥ ( x )) .
As with the CSS and OvA losses, we may parameterise¯ f y 0 ( x ) =
w > y
Φ( x ) for shared embedding Φ , and thus induce information sharing amongst all logits.
The above post-hoc approach crucially relies on estimating the expert’s error rate.
A potentiallysimpler task is learning to predict whether it is beneﬁcial to invoke the expert, i.e., whether the expert’serror is lower than that of the base model.
We now consider an alternate strategy that implements this.
4.2 Post-hoc rejector training
Our second post-hoc strategy revisits the fundamental risk (1) underpinning the learning to deferproblem.
While deﬁned in terms of a classiﬁer ¯ h : X → Y ∪ {⊥} , we may rewrite this in terms of a 200 base classiﬁer h : X → Y and rejector r : X → R , with the latter denoting the conﬁdence in deferring:
[Equation]
(13) where c mod ( x, y )
=
1 ( y 6 = h ( x )) .
Given a scorer ¯ f : X → R L +1 , we may set h ( x ) =argmax y
0 ∈
Y ¯ f y 0 ( x ) and r ( x ) =
¯ f ⊥ ( x ) − max y 0 ∈
Y ¯ f y 0 ( x ) .
Both CSS and OvA construct surrogate losses that are applied jointly over all logits ¯ f 1 , . .
, ¯ f L , ¯ f ⊥ ; equally, they construct a jointsurrogate loss for both the base classiﬁer h and rejector r .
As established earlier, the solution obtainedfrom these losses is reasonable under ﬁxed deferral cost c 0 = 0 , but tends to suﬀer when c 0 > 0 .
We may resolve this issue via a two-step procedure (see Figure 3): in the ﬁrst step, we obtain standardlogits ¯ f 1 ( x ) , . .
, ¯ f L ( x ) by minimising either the CSS or OvA loss with c 0 = 0 .
In the second step,we train only the deferral logit, by constructing a partial surrogate for only the rejector r ( x ) in (13).
Concretely, for a binary surrogate loss φ : R → R + , we may minimise the following surrogate for r :
[Equation]
With this loss, c 0 only plays a role in the c exp ( x, y ) weighting term, and thus only inﬂuences thedeferral decision; it does not induce any underﬁtting of the standard logits ¯ f 1 , . .
, ¯ f L .
Observe alsothat c mod ( x, y ) will involve the frozen logits ¯ f 1 , . .
, ¯ f L obtained from the ﬁrst stage.
We may furtherleverage these logits by parameterising r ( x ) =
¯ f ⊥ ( x ) − max y 0 ∈
Y ¯ f y 0 ( x ) , and optimising only the¯ f ⊥ ( x ) term.
In practice, exploiting the probabilities from the ﬁrst stage model can oﬀer a closerapproximation to the Bayes-optimal classiﬁer; see Appendix F for more discussion.
Theoretically, the surrogate loss in (14) provides a calibrated deferral rule for any base classiﬁer h .
For strictly proper composite φ with inverse link ψ : R → [0 , 1] , the optimal rejector is as follows.
Lemma 3.
The Bayes-optimal rejector for the loss (14) is ψ ( r ∗ ( x )) =
[Equation]
[Figure] We remark here that one may naturally iterate the above procedure multiple times, so that the deferraldecisions after the second stage inﬂuence the learning of the standard logits.
Concretely, observe thatfrom (13), it is only necessary to minimise c mod ( x, y ) on samples for which r ( x ) < 0 .
4.3 Discussion and extensions
Relation to existing work .
Post-hoc threshold correction generalises Chow’s rule [7] for learning toreject, wherein c exp ( x, y )
=
c 0 .
It also generalises the approaches of Raghu et al.
[26], Wilder et al.[36], Bansal et al.
[1] for learning to defer: these assume c 0 = 0 , and use separate models to estimateπ ( x ) and (cid:15) ( x ) .
By contrast, we estimate these quantities with a single neural model, by minimisingthe SOvA loss in (12) with ¯ f y 0 ( x ) =
w > y
Φ( x ) for y 0 ∈ Y ∪ {⊥} .
As with the standard OvA loss, thisinduces implicit information sharing between the defer logit ¯ f ⊥ and the standard logits ¯ f 1 , . .
, ¯ f L ;thus, the estimates of the expert error probability will be inﬂuenced by the base model.
Note that theconﬁdence approach of Raghu et al.
[26] was shown to be outperformed by the CSS and OvA lossesin Mozannar and Sontag [18], Verma and Nalisnick [33]; our proposed threshold correction furtherachieves competitive or superior performance to these losses (§5).
Verma and Nalisnick [33, Appendix E] considered a post-hoc scheme for the setting where thereis a hard constraint on the expert cost.
Here, one trains a probabilistic model ¯ p using the OvA loss,and defers on the samples with the largest values of ¯ p ⊥ ( x ) − max y
∈
Y ¯ p y 0 ( x ) .
While derived for 238 a diﬀerent setting, this is similar to (11), as varying the cost c 0 eﬀectively changes the fraction ofsamples that are deferred.
Note however that we use the SOvA rather than OvA loss to obtain π, (cid:15) ,which can have a notable impact on performance when L is large (see Appendix J).
Cortes et al.
[8] proposed a joint surrogate over r, ¯ f for (13) in the special case of constantc exp ( x, y )
=
c 0 , and a scalar scorer ¯ f : X → R for binary labels.
By contrast, the surrogate loss (14)handles multi-class ¯ f , generic cost functions c exp , and provides a partial surrogate over only r .
Adaptive inference and multiple experts .
Post-hoc training is naturally amenable to settings wherethere are multiple experts.
Concretely, suppose there are K experts with logits ¯ f (1) ( x ) , . .
, ¯ f ( K ) ( x ) ,and our goal is to learn which expert to employ for a given sample.
One may learn multiple deferralfunctions ¯ f (1) ⊥ ( x ) , . . .
, ¯ f ( K ) ⊥
( x ) in a post-hoc manner, and choose the smallest k with ¯ f ( k ) ⊥ ( x ) < 0 .
This setting arises naturally in adaptive inference [12], where our goal is to reduce the inference costof a learning system by allowing for variable compute across samples: intuitively, we would liketo spend less computation on “easy” samples.
To achieve this, one may construct several models(“experts”) of increasing inference cost, and defer to a suitable expert.
Such models include early-exitclassiﬁcation heads at intermediate nodes of a network [23, 31, 35, 14, 28], or sequential cascades ofmodels of increasing complexity [3, 29, 34].
Conceptually, adaptive inference involves the same core problem as learning to defer: at any givenintermediate model, one has the option to defer prediction to the next model, at the expense ofincreasing the overall computation cost.
Interestingly, similar ideas to post-hoc training have beenexplored in adaptive inference [32].
Compared to this work, we present a uniﬁed view of the learningto defer and adaptive inference problems, demonstrate limitations of existing losses for the former,and leverage generic logits ¯ f 1 , . .
, ¯ f L from any base classiﬁer.
Experimental results
We now present empirical results illustrating the eﬃcacy of both our proposed posthoc estimators.
[Figure]
5.1 Results on synthetic data
We begin with a synthetic problem where X ⊂ R 2 .
We consider a distribution P ( x, y ) =
P a ∈{ 0 , 1 } P ( x, y, a ) , where a ∈ { 0 , 1 } denotes some latent subgroup.
Figure 4 shows a sample draw from P ( x, y ) : in a nutshell, P ( x, y | a = 0) is a mixture of isotropic Gaussians, while
P ( x, y | a = 1) comprises two concentric circles.
See Appendix H.1 for a precise speciﬁcation.
Weconsider learners that have access to a sample { ( x n , y n ) }
Nn =1 from P ( x, y ) , and to expert model predictions { h exp ( x n ) } Nn =1 .
The expert is taken to be a linear classiﬁer that is trained on only samples witha
= 1 , using quadratic features: i.e., for x = ( x 1 , x 2 ) , we construct Φ( x ) =
( x 1 , x 2 , x 21 , x 22 , x 1 · x 2 ) .
We set the expert deferral cost function to be c exp ( x, y ) =
c 0 +
y 6 = h exp ( x )) .
We train linear classiﬁers using conﬁdence thresholding based on a constant deferral cost c 0 ( Chow ),the CSS and OvA losses, and our proposed post-hoc approaches.
For each learner, we vary theﬁxed deferral cost c 0 from { 0 . 0 , 0 . 05 , 0 . 10 , . . .
For each setting, we compute the fraction ofsamples deferred to the expert, and the overall accuracy of the base model plus expert.
We conduct250 independent trials, each with a diﬀerent draw of the training sample and expert predictions.
We highlight a few trends from the results in Figure 4.
First, Chow strongly underperforms post-hocthresholding when many samples are deferred.
This is because the latter does not consider the expert’serror (which is high on a = 1 ) when deferring.
Second, post-hoc training is competitive or superior to 279 CSS and OvA at most operating points.
Third, post-hoc thresholding underperforms post-hoc training.
This is by construction, as it is challenging to reliably estimate P ( y | x ) with a linear model.
However,we shall now see that on real-world benchmarks, thresholding is far more competitive.
5.2 Results on real-world data
We now report results on the CIFAR-10, CIFAR-100 [15], and ImageNet [9] datasets.
Learning to defer .
Inspired by Mozannar and Sontag [18, Section 6.2], we consider a setting wherethere is a “specialist” expert: we train an expert model on only samples with label belonging tosome subset Y sub .
Intuitively, this model “specialises” to only samples from Y sub .
We then considera base model that is trained on all samples, but is allowed to defer to the “specialist” expert.
On
CIFAR-10 and CIFAR-100, we use a ResNet-8 base model and ResNet-56 expert; on ImageNet, weuse a MobileNet-v2 base model and an EﬃcientNet-B0 expert.
The label subset Y sub comprises theﬁrst 5 labels on CIFAR-10, the 50 labels corresponding to the ﬁrst 10 “coarse labels” on CIFAR-100,and all labels corresponding to the “dog” synset on ImageNet.
We consider a cost c exp ( x, y )
=
c 0 +
y 6 = h exp ( x )) of deferring to the expert model.
Per Figures 2and 3, we apply posthoc threshold correction on top of the SOvA solution with c 0 = 0 , and posthoctraining on top of the CSS solution with c 0 = 0 .
Figure 5 compares all approaches as we vary theﬁxed cost c 0 , reporting the fraction of samples deferred to the expert, and the overall accuracy ofthe base model plus expert.
Our posthoc estimators oﬀer gains over the existing approaches at mostoperating points.
We particularly see gains in the regime where only a small fraction of samples aredeferred to the expert, which corresponds to larger values of c 0 .
Between the two post-hoc methods,we see that the threshold correction approach tends to have a (slight) edge.
This suggests that on thesebenchmarks, it is possible to obtain suﬃciently reliable estimates of P ( y | x ) and P ( y 6 = h exp ( x )) . [Figure] [Figure]
We reiterate that when c 0 is large, the existing CSS and OvA losses tend to degrade owing tounderﬁtting.
Indeed, their performance in this regime is signiﬁcantly lower than simply using thestudent model trained with the softmax cross-entropy (corresponding to Chow with 0% of samplesdeferred).
Note also that in comparison to the synthetic dataset, here L is larger and thus furtherexacerbates underﬁtting.
Finally, as in the synthetic dataset, Chow strongly underperforms when alarge fraction of samples are deferred to the expert; this is because it does not take into account theexpert’s error rate, and thus potentially defers when the expert performs akin to random guessing.
Adaptive inference .
We conclude with results on the related problem of adaptive inference.
Weconsider a setting where there is a cascade of two models of diﬀering inference costs, as measured inﬂoating point operations ( FLOPs ), and we seek to learn a deferral function that intelligently forwardsa small subset of samples to the latter model.
This allows for an inference scheme with a favourableaccuracy-compute tradeoﬀ. Compared to the previous section, we now train the experts on all labels.
We report results on CIFAR-100 and ImageNet, with the former using ResNet-8 and ResNet-56models, and the latter using a MobileNet-v2 and EﬃcientNet model.
As before, we consider thedeferral cost function c exp ( x, y ) =
c 0 +
y 6 = h exp ( x )) .
Figure 6 compares the FLOPs versus theoverall accuracy of the base model plus expert.
Here, the CSS and OvA methods again degradein the low FLOPs (high c 0 ) regime.
However, the Chow baseline is more competitive, since theexpert models tend to be highly accurate on all samples.
Our posthoc estimators are seen to stronglyoutperform CSS and OvA , while being competitive with or slightly superior to Chow .
Discussion and future work
While post-hoc training resolves issues identiﬁed with previous joint surrogate losses, it is naturalto ask whether there may be other joint surrogates that overcome these.
One candidate is thecost-sensitive loss of Lee et al.
[16], which is Fisher-consistent and is suitable when c 0 > 0 .
[-One may show thatunder this loss, the gap between the Bayes-optial scores for the highest scoring label and any competinglabel is O (cid:0) 
-]
Webelieve this loss to be worth further study, both for learning to defer and adaptive inference.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The paper
is primarily analytical in nature: it identiﬁes mathematical and empirical limitations of
existing methods, and proposes new methods for a problem where a model can query an “expert”s.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments
multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[No] The newly proposed methods in
this paper involve simple post-hoc operations over existing models.
Thus, they do not
add signiﬁcant computational overhead.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A
[Table]
) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or [Table] content?
[N/A
[Table]
[Table]
If you used crowdsourcing or conducted research with human subjects..
[Table]
) Did you [Table] the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/ [Table] [Table] you
[Table]
any [Table] risks, with
[Table]
) approvals, if applicable
[Table] [N/
[Table]
Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A [Table]