Post-hoc Estimators for Learning to Defer to an Expert
Abstract
Many practical settings allow a classiﬁer to defer predictions to one or more costly experts .
For example, the learning to defer paradigm allows a classiﬁer to defer to a human expert, at some monetary cost.
Similarly, the adaptive inference paradigm allows a base model to defer to one or more large models, at some computational cost.
The goal in these settings is to learn classiﬁcation and deferral mechanisms to optimise a suitable accuracy-cost tradeo  .
To achieve this, a central issue studied in prior work is the design of a coherent loss function for both mechanisms.
In this work, we demonstrate that existing losses can underﬁt the training set when there is a non-trivial deferral cost, owing to an implicit application of a high level of label smoothing.
To resolve this, we propose two post-hoc estimators that ﬁt a deferral function on top of a base model, either by threshold correction, or by learning when the base model’s error rate exceeds the cost of deferring to the expert.
Both approaches are equipped with theoretical guarantees, and empirically yield e  ective accuracy-cost tradeo  s on learning to defer and adaptive inference benchmarks.
Introduction
Supervised classiﬁcation conventionally considers learning a single model with good average-case test performance.
However, many practical settings allow for choosing amongst multiple models to classify a given sample.
For example, in the learning to defer ( L2D ) paradigm [18], a base classiﬁer may be used in conjunction with a human expert .
Speciﬁcally, the classiﬁer has the option of deferring to the expert, at the expense of incurring some additional cost (e.g., monetary expense).
The goal is to learn a classiﬁer and deferral function achieving a suitable accuracy-cost tradeo  .
This problem has received a surge of interest [26, 19, 22, 34] given the increasing use of machine learning in high-stakes decision making.
Interestingly, similar considerations are also found in the problem of adaptive inference for resource-constrained prediction [13]: here, a base model can choose to defer to one or more experts, which are themselves learning models with higher capacity and inference cost.
Similar to L2D, the goal is to achieve a suitable tradeo  between accuracy and (inference) cost.
A key problem in such settings is deciding when the base model should defer its prediction to the expert(s).
An intuitive strategy is to defer when the model has low prediction conﬁdence (e.g., classiﬁcation margin).
While theoretically grounded [8], this strategy can under-perform when the base model has low capacity [9].
An alternate strategy is to explicitly formulate and optimise a joint loss function for the base classiﬁer and deferral function.
This has been followed with success in the
L2D literature, particularly with the cost-sensitive softmax cross-entropy loss ( CSS ) of Mozannar and
Sontag [19], and one-versus-all ( OvA ) loss of Verma and Nalisnick [34].
While the above losses have proven e  ective they are not without limitations.
Indeed, both focus on the setting where the cost of deferring is only given by the probability that the expert makes a mistake on a sample; this does not consider the additional ﬁxed cost of querying an expert in the ﬁrst place.
As we shall see (§3) incorporating any non-zero ﬁxed cost makes these losses prone to underﬁtting ; cf.
Figure 1.
This behaviour is owing to the fact that the losses implicitly apply a high level of label smoothing .
The setting of non-zero ﬁxed cost is particularly important in adaptive inference settings, where querying an expert inherently involves paying an increased computational cost.
In this work, we resolve this issue via deferral schemes based on post-hoc estimators .
In a nutshell, we follow a two-step procedure wherein we ﬁt a deferral function on top of a base model, either by threshold correction, or by learning to predict when the base model’s error rate exceeds the deferral cost.
This allows us to build on top of the standard (zero ﬁxed cost) CSS and OvA solutions, resulting in an empirically e  ective strategy with theoretical guarantees.
In sum, our contributions are
: (i) we identify limitations in prior losses for L2D, particularly when the cost of deferring to an expert involves a non-zero ﬁxed cost in addition to the probability of the expert making a mistake (§3) (ii) we propose new post-hoc estimators for learning a deferral function, which overcome the above limitation while enjoying theoretical guarantees (§4, Theorem 2, Theorem 4) (iii) we empirically show that the new post-hoc estimators work well on synthetic and real-world benchmark datasets for both L2D and adaptive inference settings (§5).
In the course of our analysis, we additionally generalise the OvA loss to reliably account for general deferral costs ((10), Lemma 1), and to reliably estimate label probabilities in practical settings (12).
Background and Notation
Our interest is in learning settings where a classiﬁer is equipped with the option of deferring to an expert model.
For concreteness, we shall focus our exposition on the learning to defer paradigm.
2.1 Multi-class Classiﬁcation
Fix an instance space X and label space Y =
[ L ] .
= { 1 , 2 , . . .
, L } , and let P be a distribution over X ⇥ Y .
Given a training sample S =
{ ( x n , y n ) } n 2 [ N ] drawn from P , multi-class classiﬁcation seeks a classiﬁer h : X !
Y with low misclassiﬁcation error R ( h ) =
P ( y 6 = h ( x )) .
We may parameterise h as h ( x ) = argmax y 0
Y f y
( x ) for a scorer f : X !
R L .
Such a scorer may be learned by minimising the empirical surrogate risk
ˆ R ( f )
= 1 N
P n 2 [ N ] ` ( x n , y n , f ( x n )) for some loss ` : X ⇥ Y ⇥ R L !
R + , such as the softmax cross-entropy ` ( x, y, f ( x ))
= log hP y
[Equation]
x
[Equation]
i  f y ( x ) .
For binary classiﬁcation problems with L = 2 , it is common to enforce f 2 ( x ) =
f 1 ( x ) and set ` ( x, y, f ( x ))
=
( f y ( x )) for some margin loss  : R !
R + , such as  ( z )
= log(
+ e  z ) .
2.2 Learning to Defer (L2D) Problem
In the learning to defer ( L2D ) problem [18], one seeks a classiﬁer that can either make a standard prediction in Y via some base model, or defer its prediction to an expert model h exp : X !
Y , at the expense of incurring a sample-dependent cost c exp ( x, y ) > 0 .
Typically, c exp takes into account both the ﬁxed cost of deferring to the expert, which we will assume to be a constant c 0 2 [0 , 1] (e.g., normalised monetary cost), and an estimate of the probability that the expert makes a mistake on the example.
Canonical examples include c (1) exp ( x, y ) =
c 0 and c (2) exp ( x, y )
=
c 0 +
y 6 = h exp ( x )) .
The choice c (1) exp ( x, y ) corresponds to learning with a reject option or abstention [8, 2, 9, 6].
Given a suitable c exp ( x, y ) , the learning problem may be formalised as follows.
Consider a classiﬁer ¯ h : X !
Y [ { ? } equipped with a “defer” option ? .
Our goal is to minimise
[Equation]
Intuitively, when ¯ h chooses not to defer, we incur the usual misclassiﬁcation error; otherwise, we incur the cost c exp ( x, y ) of invoking the expert h exp .
One can cast (1) as an instantiation of cost-sensitive learning [11]; see Appendix B. We remark that Okati et al.
[22] considered a slightly di  erent setting, wherein one enforces a hard constraint on the expert cost c exp ( x, y ) .
We focus on a soft additive penalty in (1), but we may handle hard constraints via standard Lagrangian theory; see Appendix G.
In practice, it is useful to parameterise ¯ h in terms of a scorer ¯ f : X !
R L +1 with label logits { ¯ f 1 ( x ) , . . .
, ¯ f L ( x ) } , and “defer” logit ¯ f ?
( x ) .
We then predict via
[Equation]
In practice, we may parameterise ¯ f via neural network logits: for any y 0 2 Y [ { ? } , ¯ f y 0 ( x ) =
w > y 0  ( x ) for weights w y 0 and embedding  .
This parameterisation induces implicit sharing amongst all logits.
One conceptually useful quantity is the Bayes-optimal ¯
h ⇤ =
argmin ¯ h : X !
Y [ { ? } R def (¯ h ) , which is
[Equation]
Intuitively, we defer i  the expected cost of querying the expert is smaller than the expected error of the base model.
For the constant cost function c (1) exp ( x, y ) , this is known as Chow’s rule [8].
2.3 Losses for Learning to Defer
We review some popular recent strategies to learn ¯ f (and thus ¯ h ) below.
Conﬁdence thresholding .
For c exp ( x, y )
=
c 0 , one strategy is to learn ¯ f 1 , . .
, ¯ f L by minimising the softmax cross-entropy, and ﬁx ¯ f ?
( x ) = 1  c 0 .
This conﬁdence thresholding approach [21] mimics the Bayes-optimal classiﬁer (3).
A related series of approaches [27, 38, 1] make deferral decisions by comparing the base model conﬁdence with the conﬁdence in the expert being correct.
When the base model has low-capacity, such approaches may be sub-optimal as the base model may sub-optimally allocate its ﬁnite capacity by classifying examples that the expert can easily predict.
Cost-sensitive softmax cross-entropy ( CSS ) .
To overcome the limitations of conﬁdence thresholding, it is desirable to explicitly learn ¯ f ? as well.
To this end, Mozannar and Sontag [19] proposed a costsensitive softmax cross-entropy ( CSS ) loss: for a constant upper bound c max  1 on the classiﬁcation costs, this can be written as:
[Equation]
where ¯ p y ( x ) = exp( ¯ f y ( x )) /
P y 0 2 Y [ { ? } exp( ¯ f y 0 ( x )) is the softmax distribution over the standard and the defer label.
For c exp ( x, y )
=
c 0 +
y 6 = h exp ( x )) , we have c max = 1 + c 0 .
The loss ` CSS is
Fisher-consistent [19, Proposition 1], with Bayes-optimal prediction
[Equation]
The original loss of Mozannar and Sontag [19] uses a slightly tighter formulation, with an upper bound c max ( x, y )  1 that is sample-dependent.
Consequently, we can set c max ( x, y )
=
max { 1 , c exp ( x, y ) } .
For simplicity, we use a constant upper bound for all the losses in this paper.
One-versus-all ( OvA ) .
Despite the Fisher consistency of ` CSS , Verma and Nalisnick [34] observed that empirically, the loss may not produce calibrated estimates of the expert’s error probability.
Verma and
Nalisnick thus proposed a one-versus-all ( OvA ) loss for the cost function c exp ( x, y )
=
1 ( y 6 = h exp ( x )) :
[Equation]
Here,  : R !
R
+ is a binary proper composite loss [28], e.g.,  ( z )
= log(
+ e  z ) .
Such losses have an associated inverse link function : R !
[0 , 1] , and produce Bayes-optimal scorer
[Equation]
Intuitively, both (4) and (6) seek to have the ﬁrst L logits { ¯ f 1 ( x ) , . .
, ¯ f L ( x ) } estimate P ( y | x ) , and the reject logit ¯ f ?
( x ) model the cost of querying the expert.
With this, the highest scoring logit provides us with an estimate of the Bayes-optimal deferral decision (3).
Limitations of Existing Learning to Defer Losses
We now study existing losses for learning to defer more carefully, and show that they may underﬁt in an important practical setting.
Speciﬁcally, recall that a canonical choice of deferral cost function in (1) is c exp ( x, y )
=
1 ( y 6 = h exp ( x ))
+ c 0 , for constant c 0  0 .
Here, c 0 represents a ﬁxed cost of querying the expert, which is independent of whether the expert misclassiﬁes the sample.
Thesetting of non-zeroﬁxedcost c 0 > 0 isof practicalinterest; e.g., thisisintrinsictoadaptiveinference settings, wherein querying the expert model fundamentally involves increasing a computational overhead (see §4.3 for more discussion).
Thus, an ideal surrogate loss for (1) should be performant for c 0 > 0 .
Unfortunately, we now show that this does not hold for the CSS and OvA losses.
3.1 Underﬁtting of Cost-Sensitive Softmax Cross-Entropy when c 0 >
The cost-sensitive softmax cross-entropy ( CSS ) is in principle applicable for any deferral cost c exp ( x, y ) , and in particular c exp ( x, y )
=
c 0 +
y 6 = h exp ( x )) for any c 0  0 .
However, the focus in Mozannar and Sontag [19] was the case c 0 = 0 .
In fact, we now show that when c 0 > 0 , the loss can underﬁt .
To see this, note that the tightest possible maximal cost is c max = 1 + c 0 .
With this choice, (4) becomes
[Equation]
When c 0 = 0 , the second term vanishes, and we have a conic combination of the log-loss on the true and deferral labels.
When c 0 > 0 , the second term is active, and applies a form of label smoothing [31]: we allow for all labels y 0 2 Y  { y } to be potential “positives” for x , with a weight of c 0 2 [0 , 1] .
Put di  erently, we treat the ground-truth label distribution as a mixture of a one-hot distribution on y , and a uniform distribution over all labels.
As c 0 !
1  , we apply a large amount of smoothing on labels other than y ; this can be problematic.
Concretely, consider an x where the expert gives the wrong prediction, i.e., P ( y = h exp ( x ))
= 0 .
Then, the Bayes-optimal solution (5) will be
[Equation]
Observe that for c 0 > 0 , as L increases, the range of values in ¯ p ⇤ shrinks: indeed, the gap between the optimal probability for true label y and a competing label y 0 will be O (1 /L ) .
This small probability margin makes learning challenging: given a ﬁnite sample, it will be easy for the model to mistakenly swap the true label y with some competing label y 0 .
Thus, in settings where L is moderately large, and where there is a non-zero ﬁxed deferral cost c 0 > 0 , ` CSS may underperform.
Such issues have also been observed [7] for a similar loss developed for label noise [25].
As an illustration, we consider in Figure 1 a learning to defer setting comprising a ResNet-8 base model and a ResNet-32 “expert” on CIFAR-100.
We assume a deferral cost c exp ( x, y ) =
c 0 +
y 6 = h exp ( x )) , where c 0 is varied in [0 , 1] .
For each c 0 , we train the base model using (4), and report the train and test accuracy.
As c 0 increases, the base model shows signiﬁcant training accuracy degradation, i.e., it underﬁts.
The high level of label smoothing plays a role: in the right panel, we see that the entropy of the model’s predicted distribution over labels increases with c 0 .
(The entropy is normalised by log L to lie in [0 , 1] .)
See Appendix I for additional plots, and Appendix I.3 for results when c exp ( x, y )
=
c 0 .
It is worth pointing out that the original loss of Mozannar and Sontag [19], which uses a tighter sample-dependent upper bound, will not have the label smoothing term on the samples that the expert predicts correctly, but will still underﬁt on the samples that the expert is wrong on.
3.2 Underﬁtting of One-Versus-All Loss when c 0 >
The one-versus-all ( OvA ) loss of Verma and Nalisnick [34] was presented for the speciﬁc case of c 0 = 0 ; as stated, the loss is not applicable when c 0 > 0 .
We may however extend this loss to a general c exp ( x, y ) with upper bound c max  1 (see Appendix D for a more general derivation):
[Equation]
Compared to (6), we have an additional weighted sum over each  y 0 ( x ) .
As with OvA , this loss results in a coherent Bayes-optimal solution when  is a strictly proper composite loss [5, 28].
Recall that such losses are characterised by an inverse link function
[Equation]
, 1] , such as the logistic loss  ( z )
= log(
+ e  z ) with sigmoid inverse link ( v ) =
(1 + exp
(
[Equation]
)
[Equation]
Lemma 1.
Suppose  is a strictly proper composite loss with inverse link function .
Then, the one-versus-all loss in (10) is calibrated for the learning to defer risk (1) , with Bayes-optimal scorer
[Equation]
Unlike the optimal scorer for the CSS loss, here
[Equation]
Nonetheless, the highest scoring logit agrees with the Bayes-optimal classiﬁer (3) for learning to defer.
Reviewing (10), we see that when c max > 1 , we apply a form of label smoothing by additional terms of the form
P
[Equation]
)) , i.e., we e  ectively treat all y 0 6 = y as potential positives.
As with the cost-sensitive softmax entropy, this can result in underﬁtting; recall that with c exp ( x, y )
=
c 0 +
y 6 = h exp ( x )) , we have an upper bound c max = 1 + c 0 > 1 .
We empirically conﬁrm in Appendix I.2 that we indeed observe analogous underﬁtting behaviour as per the CSS loss in Figure 1.
The above shows that the CSS and OvA losses have a subtle limitation when c 0 > 0 .
This is not in conﬂict with the Fisher-consistency of these losses: the latter establishes that the asymptotic minimiser of these losses agrees with the Bayes-optimal classiﬁer (3).
Our analysis explicates that the precise form of these Bayes-optimal solutions may not be amenable to learning from a ﬁnite sample .
Since both losses seamlessly handle c 0 = 0 , a natural question is whether one can leverage this solution to help guide the one for c 0 > 0 .
We now present post-approaches that do precisely this.
Post-hoc Approaches to Learning to Defer
We now present two simple post-hoc strategies for learning to defer, which build on the CSS and OvA solutions for c 0 = 0 via threshold correction and re-training the deferral function.
4.1 Post-hoc Threshold Correction
Our ﬁrst post-hoc strategy (Figure 2) exploits the form of the Bayes-optimal classiﬁer in (3): to mimic this classiﬁer for the cost function c exp ( x, y )
=
1 ( y 6 = h exp ( x ))
+ c 0 , it su  ces to obtain estimates of P ( y | x ) and the expected deferral cost E [ c exp ( x, y )]
=
P ( y 6 = h exp ( x ))
+ c 0 .
Consequently, suppose we learn models ⇡ : X ! [0 , 1] L and ✏ : X !
[0 , 1] estimating P ( y | x ) and P ( y 6 = h exp ( x )) respectively.
Then, for any c 0  0 , we can approximate (3) via the classiﬁer
[Equation] [Equation]
[Figure]
Intuitively, this procedure ought to perform well when ⇡ ( x ) is close to ⌘ ( x ) .
=
[ P ( y | x )] y 2 Y , and ✏ ( x ) is close to P ( y 6 = h exp ( x )) .
The following excess risk bound for ¯ h formalises this intuition.
Theorem 2.
Pick any ⇡ : X !
[0 , 1] L and ✏ : X !
Let ¯ h be the corresponding classiﬁer in (11) .
Then, the excess risk of ¯ h compared to the Bayes-optimal classiﬁer ¯ h ⇤ (3) is:
[Equation]
Crucially, observe that we can learn ⇡ and ✏ once , and then sweep across di  erent c 0 values in (11).
Thus, we can attempt to leverage the solutions for either the CSS or OvA losses with c 0 = 0 — recalling that these do not exhibit underﬁtting, unlike when c 0 > 0 — and then simply vary c 0 post-hoc.
Indeed, both losses’ Bayes-optimal solutions (5), (7) for c 0 = 0 suggest that a transform of ¯ f 1 , . . .
, ¯ f L provide estimates of P ( y | x ) , while a transform of ¯ f ? provides an estimate of P ( y 6 = h exp ( x )) .
Unfortunately, in practice, both losses are potentially suboptimal to learn ⇡ , ✏ .
For the CSS loss, Verma and Nalisnick [34] showed it does not provide calibrated estimates of P ( y 6 = h exp ( x )) .
On the other hand, while the OvA loss provides calibrated estimates of P ( y 6 = h exp ( x )) , its probability estimates of P ( y | x ) may underperform when L is large (see Appendix J).
We overcome this with a simple solution: we estimate ⇡ via the softmax cross-entropy, and ✏ via the OvA loss.
This is achieved via a hybrid softmax cross-entropy plus OvA loss ( SOvA ):
[Equation]
where p y ( x ) = exp( ¯ f y ( x )) /
P y 0
Y exp( ¯ f y 0 ( x )) .
Following the Bayes-optimal solution, we set ⇡ ( x ) =
( p 1 ( x ) , . . .
, p L ( x )) and ✏ ( x ) = ( ¯ f ? ( x )) .
As with the CSS and OvA losses, we may parameterise ¯ f y 0 ( x ) =
w > y 0  ( x ) for shared  , and thus induce information sharing amongst all logits.
The above post-hoc approach crucially relies on estimating the expert’s error rate.
A potentially simpler task is learning to predict whether it is beneﬁcial to invoke the expert, i.e., whether the expert’s error is lower than that of the base model.
We now consider an alternate strategy that implements this.
4.2 Post-hoc Rejector Training
Our second post-hoc strategy revisits the fundamental risk (1) underpinning the learning to defer problem.
While deﬁned in terms of a classiﬁer ¯ h : X !
Y [ { ? } , we may rewrite this in terms of a base classiﬁer h : X !
Y and rejector r : X !
R , with the latter denoting the conﬁdence in deferring:
[Equation]
where c mod ( x, y )
=
1 ( y 6 = h ( x )) .
Given a scorer ¯ f : X !
R L +1 , we may set h ( x ) =
argmax y 0
Y ¯ f y 0 ( x ) and r ( x ) =
¯ f ?
( x )  max y 0
Y ¯ f y 0 ( x ) .
Both CSS and OvA construct surrogate losses that are applied jointly over all logits ¯ f 1 , . .
, ¯ f L , ¯ f ? ; equally, they construct a joint surrogate loss for both the base classiﬁer h and rejector r .
As established earlier, the solution obtained from these losses is reasonable under ﬁxed deferral cost c 0 = 0 , but tends to su  er when c 0 > 0 .
We may resolve this issue via a two-step procedure (see Figure 3): in the ﬁrst step, we obtain standard logits ¯ f 1 ( x ) , . .
, ¯ f L ( x ) by minimising either the CSS or OvA loss with c 0 = 0 .
In the second step, we train only the deferral logit, by constructing a partial surrogate for only the rejector r ( x ) in (13).
Concretely, for a binary surrogate loss  : R !
R + , we may deﬁne the following surrogate for r :
[Equation] [Equation]
and minimize the resulting partial surrogate risk over r :
[Equation] [Equation]
With this loss, c 0 only plays a role in the c exp ( x, y ) weighting term, and thus only inﬂuences the deferral decision; it does not induce any underﬁtting of the standard logits ¯ f 1 , . .
, ¯ f L .
Observe also
[Figure]
that c mod ( x, y ) will involve the frozen logits ¯ f 1 , . .
, ¯ f L obtained from the ﬁrst stage.
We may further leverage these logits by parameterising r ( x ) =
¯ f ?
( x )  max y 0
Y ¯ f y 0 ( x ) , and optimising only the ¯ f ?
( x ) term.
In practice, exploiting the probabilities from the ﬁrst stage model can o  er a closer approximation to the Bayes-optimal classiﬁer; see Appendix F for more discussion.
Theoretically, the surrogate loss in (14) provides a calibrated deferral rule for any base classiﬁer h .
For strictly proper composite  with inverse link : R !
[0 , 1] , the optimal rejector is as follows.
Lemma 3.
The Bayes-optimal rejector for the loss (14) is ( r ⇤ ( x )) =
[Equation]
Note that when the ﬁrst stage logits agree with the Bayes-optimal solution, E y | x [ c mod ( x, y )]
= 1  max y 2 Y P
( y | x ) , and so deferring when ( r ⇤ ( x )) > 0 .
5 produces the optimal decision.
We next provide a bound an excess risk bound for a classiﬁer learned using the post-hoc training procedure, under the assumption that the surrogate loss  used is classiﬁcation calibrated [3].
Theorem 4.
Suppose  is classiﬁcation calibrated and c exp ( x, y )
[ c min , c max ] for some c min > 0 .
Denote the misclassiﬁcation risk for classiﬁer h by R err ( h ) =
E [ 1 ( y 6 = h ( x ))] .
Let ˆ r be the rejector obtained by minimizing the partial surrogate risk in (15) for base classiﬁer ˆ h .
Then the excess L2D risk for the resulting classiﬁer (ˆ h, ˆ r ) is bounded in terms of the excess surrogate risk for the rejector ˆ r and the excess misclassiﬁcation risk for the base classiﬁer ˆ h :
[Equation]
for some increasing function
[Equation]
4.3 Discussion and Extensions
Relation to existing work .
Post-hoc threshold correction generalises Chow’s rule [8] for learning to reject, wherein c exp ( x, y )
=
c 0 .
It also generalises the approaches of Raghu et al.
[27], Wilder et al.
[37], Bansal et al.
[1] for learning to defer: these assume c 0 = 0 , and use separate models to estimate ⇡ ( x ) and ✏ ( x ) .
By contrast, we estimate these quantities with a single neural model, by minimising the SOvA loss in (12) with ¯ f y
( x ) = w > y 0  ( x ) for y 0 2 Y [ { ? } .
As with the standard OvA loss, this induces implicit information sharing between the defer logit ¯ f ?
and the standard logits ¯ f 1 , . .
, ¯ f L ; thus, the estimates of the expert error probability will be inﬂuenced by the base model.
Note that the conﬁdence approach of Raghu et al.
[27] was shown to be outperformed by the CSS and OvA losses in Mozannar and Sontag [19], Verma and Nalisnick [34]; our proposed threshold correction further achieves competitive or superior performance to these losses (§5).
Verma and Nalisnick [34, Appendix E] considered a post-hoc scheme for the setting where there is a hard constraint on the expert cost.
Here, one trains a probabilistic model ¯ p using the OvA loss, and defers on the samples with the largest values of ¯ p ?
( x )  max y 0
Y ¯ p y 0 ( x ) .
While derived for a di  erent setting, this is similar to (11), as varying the cost c 0 e  ectively changes the fraction of samples that are deferred.
Note however that we use the SOvA rather than OvA loss to obtain ⇡ , ✏ , which can have a notable impact on performance when L is large (see Appendix J).
Cortes et al.
[9] proposed a joint surrogate over r, ¯ f for (13) in the special case of constant c exp ( x, y )
=
c 0 , and a scalar scorer ¯ f : X !
R for binary labels.
By contrast, the surrogate loss (14) handles multi-class ¯ f , generic cost functions c exp , and provides a partial surrogate over only r .
Adaptive inference and multiple experts .
Post-hoc training is naturally amenable to settings where there are multiple experts.
Concretely, suppose there are K experts with logits ¯ f (1) ( x ) , . .
, ¯ f ( K ) ( x ) , and our goal is to learn which expert to employ for a given sample.
One may learn multiple deferral functions ¯ f (1) ?
( x ) , . . .
, ¯ f ( K ) ?
( x ) in a post-hoc manner, and choose the smallest k with ¯ f ( k ) ?
( x ) < 0 .
This setting arises naturally in adaptive inference [13], where our goal is to reduce the inference cost of a learning system by allowing for variable compute across samples: intuitively, we would like to spend less computation on “easy” samples.
To achieve this, one may construct several models (“experts”) of increasing inference cost, and defer to a suitable expert.
Such models include early-exit classiﬁcation heads at intermediate nodes of a network [24, 32, 36, 15, 29], or sequential cascades of models of increasing complexity [4, 30, 35].
Conceptually, adaptive inference involves the same core problem as learning to defer: at any given intermediate model, one has the option to defer prediction to the next model, at the expense of increasing the overall computation cost.
Interestingly, similar ideas to post-hoc training have been explored in adaptive inference [33].
Compared to this work, we present a uniﬁed view of the learning to defer and adaptive inference problems, demonstrate limitations of existing losses for the former, and leverage generic logits ¯ f 1 , . .
, ¯ f L from any base classiﬁer.
Experimental Results
We now present empirical results illustrating the e  cacy of both our proposed posthoc estimators.
5.1 Results on Synthetic Data
We begin with a synthetic problem where X ⇢ R 2 .
We consider a distribution P ( x, y ) =
P a 2 { 0 , 1 } P ( x, y, a ) , where a 2 { 0 , 1 } denotes some latent subgroup.
Figure 4 shows a sample draw from P ( x, y ) : in a nutshell, P ( x, y | a = 0) is a mixture of isotropic Gaussians, while
P ( x, y | a = 1) comprises two concentric circles.
See Appendix H.1 for a precise speciﬁcation.
We consider learners that have access to a sample { ( x n , y n ) }
Nn =1 from P ( x, y ) , and to expert model predictions { h exp ( x n ) } Nn =1 .
The expert is taken to be a linear classiﬁer that is trained on only samples with a = 1 , using quadratic features: i.e., for x = ( x 1 , x 2 ) , we construct  ( x ) =
( x 1 , x 2 , x 21 , x 22 , x 1 · x 2 ) .
We set the expert deferral cost function to be c exp ( x, y ) =
c 0 +
y 6 = h exp ( x )) .
We train linear classiﬁers using conﬁdence thresholding based on a constant deferral cost c 0 ( Chow ), the CSS and OvA losses, and our proposed post-hoc approaches.
For each learner, we vary the ﬁxed deferral cost c 0 from { 0 . 0 , 0 . 05 , 0 . 10 , . . .
For each setting, we compute the fraction of samples deferred to the expert, and the overall accuracy of the base model plus expert.
We conduct 250 independent trials, each with a di  erent draw of the training sample and expert predictions.
We highlight a few trends from the results in Figure 4.
First, Chow strongly underperforms post-hoc thresholding when many samples are deferred.
This is because the latter does not consider the expert’s error (which is high on a = 1 ) when deferring.
Second, post-hoc training is competitive or superior to CSS and OvA at most operating points.
Third, post-hoc thresholding underperforms post-hoc training.
This is by construction, as it is challenging to reliably estimate P ( y | x ) with a linear model.
However, we shall now see that on real-world benchmarks, thresholding is far more competitive.
[Table]
(cid:51)(cid:82)(cid:86)(cid:87)(cid:16)(cid:75)(cid:82)(cid:70)(cid:3)7(cid:75)(cid:85)(cid:72)(cid:86)(cid:75)(cid:82)O(cid:71)(cid:76)(cid:81)(cid:74)
5.2 Results on Real-World Data
We now report results on the CIFAR-10, CIFAR-100 [16] and ImageNet [10] datasets.
Learning to defer .
Inspired by Mozannar and Sontag [19, Section 6.2], we consider a setting where there is a “specialist” expert: we train an expert model on only samples with label belonging to some subset Y sub .
Intuitively, this model “specialises” to only samples from Y sub .
We then consider a base model that is trained on all samples, but is allowed to defer to the “specialist” expert.
On
CIFAR-10 and CIFAR-100, we use a ResNet-8 base model and ResNet-56 expert; on ImageNet, we use a MobileNet-v2 base model and an E  cientNet-B0 expert.
The label subset Y sub comprises the ﬁrst 5 labels on CIFAR-10, the 50 labels corresponding to the ﬁrst 10 “coarse labels” on CIFAR-100, and all labels corresponding to the “dog” synset on ImageNet.
We consider a cost c exp ( x, y )
=
c 0 +
y 6 = h exp ( x )) of deferring to the expert model.
Per Figuresand 3, we apply posthoc threshold correction on top of the SOvA solution with c 0 = 0 , and posthoc training on top of the CSS solution with c 0 = 0 .
Figure 5 compares all approaches as we vary the ﬁxed cost c 0 , reporting the fraction of samples deferred to the expert, and the overall accuracy of the base model plus expert.
Our posthoc estimators o  er gains over the existing approaches at most operating points.
We particularly see gains in the regime where only a small fraction of samples are deferred to the expert, which corresponds to larger values of c 0 .
Between the two post-hoc methods, we see that the threshold correction approach tends to have a (slight) edge.
This suggests that on these benchmarks, it is possible to obtain su  ciently reliable estimates of P ( y | x ) and P ( y 6 = h exp ( x )) .
We reiterate that when c 0 is large, the existing CSS and OvA losses tend to degrade owing to underﬁtting.
Indeed, their performance in this regime is signiﬁcantly lower than simply using the student model trained with the softmax cross-entropy (corresponding to Chow with 0% of samples deferred).
Note also that in comparison to the synthetic dataset, here L is larger and thus further exacerbates underﬁtting.
Finally, as in the synthetic dataset, Chow strongly underperforms when a large fraction of samples are deferred to the expert; this is because it does not take into account the expert’s error rate, and thus potentially defers when the expert performs akin to random guessing.
Adaptive inference .
We conclude with results on the related problem of adaptive inference.
We consider a setting where there is a cascade of two models of di  ering inference costs, as measured in ﬂoating point operations ( FLOPs ), and we seek to learn a deferral function that intelligently forwards a small subset of samples to the latter model.
This allows for an inference scheme with a favourable accuracy-compute tradeo  .
Compared to the previous section, we now train the experts on all labels.
We report results on CIFAR-100 and ImageNet, with the former using ResNet-8 and ResNetmodels, and the latter using a MobileNet-v2 and E  cientNet model.
As before, we consider the deferral cost function c exp ( x, y ) =
c 0 +
y 6 = h exp ( x )) .
Figure 6 compares the FLOPs versus the overall accuracy of the base model plus expert.
Here, the CSS and OvA methods again degrade in the low FLOPs (high c 0 ) regime.
However, the Chow baseline is more competitive, since the expert models tend to be highly accurate on all samples.
Our posthoc estimators are seen to strongly outperform CSS and OvA , while being competitive with or slightly superior to Chow .
[Table]
(cid:26)(cid:19) (cid:26)(cid:24) (cid:36) FF (cid:88) (cid:85) (cid:68)
F (cid:92) (cid:3) (cid:11) (cid:8) (cid:12) (cid:38)(cid:75)(cid:82)(cid:90) (cid:38) (cid:50)(cid:89)(cid:36) (cid:51)(cid:82)(cid:86)(cid:87)(cid:16)(cid:75)(cid:82)F(cid:3)7(cid:85)(cid:68)L(cid:81)L(cid:81)(cid:74) (cid:51)(cid:82)(cid:86)(cid:87)(cid:16)(cid:75)(cid:82)F(cid:3)7(cid:75)(cid:85)(cid:72)(cid:86)(cid:75)(cid:82)O(cid:71)L(cid:81)(cid:74)
Discussion and Future Work
While post-hoc training resolves issues identiﬁed with previous joint surrogate losses, it is not without limitation: it relies crucially on reliable base classiﬁers from the ﬁrst phase.
As formalised in
Theorem 2, any degradation in performance for the latter is also transferred to the ﬁnal post-hoc predictor.
As this may be further compounded with estimation errors in the second phase, it is natural to ask whether there may be other joint surrogates that do not exhibit underﬁtting.
One candidate is the cost-sensitive loss of Lee et al.
[17], which is Fisher-consistent and is suitable when c 0 > 0 .
This loss imposes the constraint that
[Equation]
0 , and takes the form ` ( x, y, ¯ f ( x ))
=
[Equation]
i
[Equation]
One may show that under this loss, the gap between the Bayes-optial scores for the highest scoring label and any competing label is O
[Equation]
, and thus is not adversely a  ected as L increases.
As a qualifying comment, the loss poses challenges due to being non-di  erentiable, and requiring constraints on the logits.
Nonetheless, further study of the viability of this loss would be worthwhile.
Acknowledgment.
We thank Hussein Mozannar for pointing out that the original cost-sensitive softmax cross-entropy loss of Mozannar and Sontag [19] uses a slightly tighter formulation than the one we consider, and will not underﬁt on samples that the expert predicts correctly.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The paper is primarily analytical in nature: it identiﬁes mathematical and empirical limitations of existing methods, and proposes new methods for a problem where a model can query an “expert”s.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] The newly proposed methods in this paper involve simple post-hoc operations over existing models.
Thus, they do not add signiﬁcant computational overhead.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or o  ensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]