P REDICT R ESPONSIBLY : I NCREASING F AIRNESS BY L EARNING TO D EFER
Anonymous authors
Paper under double-blind review
A BSTRACT
When machine learning models are used for high-stakes decisions, they should predict accurately, fairly, and responsibly. To fulﬁll these three requirements, a model must be able to output a reject option (i.e. say “I Don’t Know”) when it is not qualiﬁed to make a prediction.
In this work, we propose learning to defer , a method by which a model can defer judgment to a downstream decision-maker such as a human user. We show that learning to defer generalizes the rejection learning framework in two ways: by considering the effect of other agents in the decision-making process, and by allowing for optimization of complex objectives.
We propose a learning algorithm which accounts for potential biases held by decision-makers later in a pipeline.
Experiments on real-world datasets demonstrate that learning to defer can make a model not only more accurate but also less biased.
Even when operated by highly biased users, we show that deferring models can still greatly improve the fairness of the entire pipeline.
I NTRODUCTION
Recent machine learning advances have increased our reliance on learned automated systems in complex, high-stakes domains such as loan approvals (Burrell, 2016), medical diagnosis (Esteva et al., 2017), and criminal justice (Kirchner et al., 2016).
This growing use of automated decisionmaking has raised questions about the obligations of classiﬁcation systems.
In many high-stakes situations, machine learning systems should satisfy (at least) three objectives: predict accurately (predictions should be broadly effective indicators of ground truth), predict fairly (predictions should be unbiased with respect to different types of input), and predict responsibly (predictions should not be made if the model cannot conﬁdently satisfy the previous two objectives).
Given these requirements, we propose learning to defer .
When deferring, the algorithm does not output a prediction; rather it says “I Don’t Know” (IDK), indicating it has insufﬁcient information to make a responsible prediction, and that a more qualiﬁed external decision-maker (DM) is required.
For example, in medical diagnosis, the deferred cases would lead to more medical tests, and a second expert opinion.
Learning to defer extends the common rejection learning framework (Chow, 1957; Cortes et al., 2016) in two ways. Firstly, it considers the expected output of the DM on each example, more accurately optimizing the output of the joint DM-model system. Furthermore, it can be used with a variety of training objectives, whereas most rejection learning research focuses solely on classiﬁcation accuracy.
We believe that algorithms that can defer, i.e., yield to more informed decision-makers when they cannot predict responsibly, are an essential component to accountable and reliable automated systems.
In this work, we show that the standard rejection learning paradigm (learning to punt ) is inadequate, if these models are intended to work as part of a larger system. We propose an alternative decision making framework (learning to defer ) to learn and evaluate these models.
We ﬁnd that embedding a deferring model in a pipeline can improve the accuracy and fairness of the pipeline as a whole , particularly if the model has insight into decision makers later in the pipeline.
We simulate such a pipeline where our model can defer judgment to a better-informed decision maker, echoing real-world situations where downstream decision makers have more resources or information.
We propose different formulations of these models along with a learning algorithm for training a model that can work optimally with such a decision-maker.
Our experimental results on two real-world datasets, from the legal and health domains, show that this algorithm learns models which, through deferring, can work with users to make fairer, more responsible decisions.
R ELATED W ORK
Notions of Fairness.
One of the most challenging aspect of machine learning approaches to fairness is formulating an operational deﬁnition.
Several works have focused on the goal of treating similar people similarly (individual fairness) and the resulting necessity of fair-awareness – showing that it may be essential to give the algorithm knowledge of the sensitive variable (Dwork et al., 2011;
Zemel et al., 2013; Dwork et al., 2017).
Some deﬁnitions of fairness center around statistical parity (Kamiran & Calders, 2009; Kamishima et al., 2012), calibration (Pleiss et al., 2017;
Guo et al., 2017) or disparate impact/equalized odds (Chouldechova, 2016;
Hardt et al., 2016; Kleinberg et al., 2016; Zafar et al., 2017).
It has been shown that disparate impact and calibration cannot be simultaneously satisﬁed (Chouldechova, 2016; Kleinberg et al., 2016).
Hardt et al.
(2016) present the related notion of “equal opportunity”.
In a subsequent paper, Woodworth et al.
(2017) argue that in practice fairness criteria should be part of the learning algorithm, not post-hoc.
Zafar et al.
(2017) and Bechavod & Ligett (2017) develop and implement learning algorithms that integrate equalized odds into learning via regularization.
Incorporating IDK.
While we are the ﬁrst to propose learning to defer, some works have examined the “I don’t know” (IDK) option (cf. rejection learning , see Cortes et al.
for a thorough survey), beginning with Chow (1957; 1970) who studies the tradeoff between error-rate and rejection rate.
Cortes et al.
(2016) develop a framework for integrating IDK directly into learning.
KWIK (Knows-What-It-Knows) learning is proposed in Li et al.
(2011) as a theoretical framework.
Attenberg et al.
(2011) discuss the difﬁculty of a model learning what it doesn’t know (particularly rare cases), and analyzes how human users can audit such models.
Wang et al.
(2017) propose a cascading model, which can be learned from end-to-end; higher levels can say IDK and pass the decision on to lower levels.
Similarly, Kocak et al.
; Cortes et al.
; Bartlett & Wegkamp (2008) provide algorithms for saying IDK in classiﬁcation and Ripley (2007) provides a statistical overview of the problem, including both “don’t know” and “outlier” options.
However, none of these works look at the fairness impact of this procedure.
A few papers have addressed topics related to both fairness and IDK.
Bower et al.
(2017) describe fair sequential decision making but do not have an IDK concept, nor do they provide a learning procedure.
In Joseph et al.
, the authors show theoretical connections between KWIK-learning and a proposed method for fair bandit learning.
Grgi-Hlaca et al.
(2017) discuss fairness that can arise out of a mixture of classiﬁers.
However, they do not provide a learning procedure, nor do they address sequential decision making, which we believe is of great practical importance. Varshney &
Alemzadeh (2017) propose “safety reserves” and “safe fail” options which combine learning with rejection and fairness/safety, but do not suggest how such options may be learned or analyze the effect of a larger decision-making framework.
AI Safety.
Finally, our work also touches on aspects of the AI safety literature - we provide a method by which a machine learns to work optimally with a human.
This is conceptually similar to work such as Milli et al.
; Soares et al.
(2015), which discuss the situations in which a robot should be compliant/cooperative with a human.
The idea of a machine and human jointly producing a fair classiﬁer also relates to Hadﬁeld-Menell et al.
(2016), which describes algorithms for machines to align with human values.
A J OINT D ECISION -M AKING F RAMEWORK
Previous works in rejection learning (see Sec. 2) have proposed models that can choose to not classify (say IDK/reject).
In these works, the standard method is to optimize the accuracy-IDK tradeoff: how much can a model improve its accuracy on the cases it does classify by saying IDK to some cases?
We ﬁnd this paradigm inadequate.
In many of the high-stakes applications this type of work is aimed at, an IDK is not the end of the story.
Rather, a decision must be made eventually on every example, whether the model chooses to classify it or not.
Say our model is trained to detect melanoma, and when it says IDK, a human doctor can run an extra suite of medical tests.
The model learns that it is very inaccurate at detecting amelanocytic (non-pigmented) melanoma, and says IDK if this might be the case.
However, suppose that the doctor is even less accurate at detecting amelanocytic melanoma than the model is.
Then, we may prefer the model to make a prediction despite its uncertainty.
Conversely, if there are some patients that the doctor knows well, then they may have a more informed, nuanced opinion than the model.
Then, we may prefer the model to say IDK more frequently relative to its internal uncertainty.
Saying IDK on the wrong examples can also have fairness consequences.
If the doctor’s decisions bias against a certain group, then it is probably preferable for our model (if it is less biased) to defer less frequently on the cases of that group.
In short, the model we train is part of a larger pipeline, and we should be training and evaluating the performance of the pipeline with this model included , rather than solely focusing on the model itself.
To enable this, we deﬁne a general two-step framework for decision-making (Fig 1).
The ﬁrst step is an automated model whose parameters we want to learn.
The second step is some external decision maker (DM) which we do not have control over; this could be a human user or a more resourceintensive tool.
The decision-making ﬂow is done as a cascade, where the ﬁrst-step model can either predict (positive/negative) or say IDK.
If it predicts, the DM trusts the model completely, and outputs that prediction.
However, if it says IDK, the DM makes its own decision.
We assume that the DM is more powerful than the model we train — reﬂecting a number of practical scenarios where decision makers later in the chain have more resources for efﬁciency, security, or contextual reasons. This system can be expressed by Y sys = sY DM + (1 − s )
Y model , where Y model , Y DM , Y sys are the model, DM, and system output respectively, and s is a binary IDK decision variable ( s =means IDK).
Suppose we want to optimize some loss function L ( Y, Y sys ) .
In learning to defer, we train s and Y model to optimize L ( Y, sY DM +(1 − s ) Y model ) . In Appendix A, we prove that learning to defer is equivalent to rejection learning for a broad class of loss functions, including classiﬁcation error, if the DM is an oracle. Since our DM is rarely an oracle, learning to defer therefore yields a modeling advantage over rejection learning. Furthermore, learning to defer allows us to optimize a variety of objectives L ( Y, Y sys ) , whereas most rejection learning research focuses on classiﬁcation error.
In the rest of this work, we show how to learn fair IDK models in this framework.
The paper proceeds as follows: in Sec.
4 we give some background on the fairness setup; in Sec.
5 we describe two methods of learning IDK models and how we may learn them in a fair way; and in Sec.
6 we give a learning algorithm for optimizing models to succeed in this framework.
In Sec.
7 we show results on two real-world datasets.
B ACKGROUND : F AIR C LASSIFICATION
In fair binary classiﬁcation, we have data X , labels Y , predictions ˆ Y , and sensitive attribute A , assuming for simplicity that Y, ˆ Y , A ∈ { 0 , 1 } .
In this work we assume that A is known (fair-aware) and that it is a single binary attribute (e.g., gender, race, age, etc.); extensions to more general settings are straightforward.
The aim is twofold: ﬁrstly, that the classiﬁer is accurate i.e., Y i
= ˆ Y i ; and secondly, that the classiﬁer is fair with respect to A i.e., ˆ Y does not discriminate unfairly against examples with a particular value of A .
Classiﬁers with fairness constraints provably achieve worse error rates (cf.
Chouldechova (2016); Kleinberg et al. (2016)).
We thus deﬁne a loss function which trades off between these two objectives, relaxing the hard constraint proposed by models like (Hardt et al., 2016) and yielding a regularizer, similar to (Kamishima et al., 2012; Bechavod & Ligett, 2017).
We use disparate impact (DI) as our fairness metric (Chouldechova, 2016), as it is becoming widely
Here we deﬁne a continuous relaxation of DI, using probabilistic output p
= P [ Y = 1] ∈ [0 , 1] :
[Equation]
Note that constraining DI = 0 is equivalent to equalized odds (Hardt et al., 2016).
If we constrain p ∈ { 0 , 1 } , we say we are using hard thresholds ; allowing p ∈
[0 , 1] is soft thresholds .
We include a hyperparameter α to balance accuracy and fairness; there is no “correct” way to weight these.
When we learn such a model, p is a function of X parametrized by θ .
Our regularized fair loss function ( L Fair , or L F ) combines cross-entropy for accuracy with this fairness metric:
[Equation]
S AYING IDK: L EARNING TO P UNT
We now discuss two model formulations that can output IDK: ordinal regression, and neural networks with weight uncertainty.
Both of these models build on binary classiﬁers by allowing them to express some kind of uncertainty.
In this section, we discuss these IDK models and how to train them to be fair; in the following section we address how to train them to take into account the downstream decision-maker.
5. O RDINAL R EGRESSION
We extend binary classiﬁers to include a third option, yielding a model that can classify examples as “positive”, “negative” or “IDK”.
This allows the model to punt , i.e., to output IDK when it prefers not to commit to a positive or negative prediction.
We base our IDK models on ordinal regression with three categories (positive, IDK, and negative).
These models involve learning two thresholds τ =
( t 0 , t 1 ) (see
Figure 2).
We can train with either hard or soft thresholds.
If soft, each threshold t i , i ∈ { 0 , 1 } is associated with a sigmoid function σ i , where σ i ( x ) = σ ( x − t i ) ; recall that σ ( x ) = [Equation]
These thresholds yield an ordinal regression, which produces three values for every score x :
P ( x ) , I ( x ) , N ( x ) ∈ [0 , 1] , s.t. P ( x ) + I ( x ) + N ( x ) = 1 .
Using hard thresholds simply restricts
P, I, N ∈ { 0 , 1 } 3 (one-hot vector).
We can also calculate a score p ( x ) ∈ [0 , 1] , which we interpret as the model’s prediction disregarding uncertainty.
These values are:
[Equation]
P represents the model’s bet that x is “positive”, N the bet that x is “negative”, and I is the model hedging its bets; this rises with uncertainty.
Note that p is minimized at P = N ; this is also where
I is maximized.
At test time, we use the thresholds to partition the examples.
On each example, the model outputs a score x ∈ R (the logit for the ordinal regression), and a prediction p .
If t 0 < x < t 1 , then we replace the model’s prediction p with IDK.
If x ≤ t 0 or x ≥ t 1 , we leave p as is.
To encourage fairness, we can learn a separate set of thresholds for each group: ( t 0 ,A =0 , t 1 ,A =0 ) and ( t 0 ,A =1 , t 1 ,A =1 ) ; then apply the appropriate set of thresholds to each example.
We can also regularize IDK classiﬁers for fairness.
When training this model, P , I , and N are parametrized functions, with model parameters θ and thresholds τ .
Using soft thresholds, the regularized loss function L FairPunt ( L FP ) is:
[Equation]
Note that we add a term penalizing I ( X ) , to prevent the trivial solution of always outputting IDK.
In addition, we regularize the disparate impact for P ( X ) and N ( X ) separately.
This was not necessary in the binary case, since these two probabilities would have always summed to 1.
We learn soft thresholds end-to-end; for hard thresholds we use a post-hoc thresholding scheme (see Appendix D).
5. B AYESIAN W EIGHT U NCERTAINTY
We can also take a Bayesian approach to uncertainty by learning a distribution over the weights of a neural network (Blundell et al., 2015).
In this method, we use variational inference to approximate the posterior distribution of the model weights given the data.
When sampling from this distribution, we can obtain an estimate of the uncertainty.
If sampling several times yields widely varying results, we can state the model is uncertain on that example.
This model outputs a prediction p and an uncertainty π for example x .
We calculate these by sampling J times from the model, yielding J predictions z j ∈ [0 , 1] .
Our prediction p is the sample mean µ = 1 J (cid:80)
Jj =1 z j .
To numerically represent our uncertainty, we can use signal-to-noise ratio, deﬁned as S = | µ − 0 .
5 | σ , based on µ and the sample standard deviation σ = (cid:113) (cid:80)
Jj {+=
+}
( z j − µ ) 2 J −.
The reciprocal of this ( π = 1 /S ) allows high values to be more uncertain, while π = σ (log(1 /S ))
(where σ is the logistic function) yields uncertainty values in a [0 , 1] range.
At test time, the system can threshold this uncertainty; any example with uncertainty beyond a threshold is punted to the
DM.
We can regularize this Bayesian model to improve fairness as in the standard binary classiﬁer.
In the likelihood term for the variational inference, we can simply add the disparate impact regularizer (Eq. 1), making solutions of low disparate impact more likely.
With weights w and variational parameters θ , our variational lower bound L V is then:
[Equation]
L EARNING TO D EFER
IDK models come with a consequence: when a model punts, the prediction is made instead by some external, possibly biased decision maker (DM) e.g., a human expert.
In this work we assume that this DM is possibly biased, but is more accurate than the model; perhaps the DM is a judge with detailed information on repeat offenders, and with more information about the defendant than the model has, or a doctor who can conduct a suite of complex medical tests.
Here we introduce a distinction between learning to punt and learning to defer .
In learning to punt, the goal is absolute: the model’s aim is to identify the examples where it has a low chance of being correct.
In learning to defer, the model has some information about the DM and takes this into account in its IDK decisions.
Hence the goal is relative: the model’s aim is to identify the examples where the DM’s chance of being correct is much higher than its own.
If the model punts mostly on cases where the DM is very inaccurate or unfair, then the joint predictions made by the model-DM pair may be poor, even if the model’s own predictions are good.
We can think of learning to punt as
DM-unaware learning , and learning to defer as DM-aware learning .
To conduct DM-aware learning, we can modify the model presented in Section 5 to take an extra input: the DM’s scores on every case in the training set.
The model is then optimized for some loss function L ( Y, A, X ) ; for our purposes, this loss will be a combination of accuracy and fairness.
We propose the following general method, drawing inspiration from mixture-of-experts (Jacobs et al., 1991).
We introduce a mixing parameter π for each example x , which is the probability of deferral ; that is, the probability that the DM makes the ﬁnal decision on the example x , rather than the model.
Then, 1 − π is the probability that the model’s decision becomes the ﬁnal output of the system.
Let s ∼ Ber ( π ) .
Our mixing parameter π corresponds to our model’s uncertainty estimate — I ( x ) in ordinal regression, σ (log( 1 S )) in the Bayesian neural network.
Let p be the ﬁrst stage model’s predictions and ˜ Y be the DM’s predictions.
We can express the joint system’s predictions ˆ p as
[Equation]
In learning this model, we can parametrize p and π by θ =
( θ p , θ π ) , which may be shared parameters.
We can then deﬁne our loss function ( L Defer , or L D ) as an expectation over the Bernoulli variables s i ∼ Ber ( π ( x i , θ π )) :
[Equation]
We call this learning to defer .
When optimizing this function, the model learns to recognize when there is relevant information that is not present in the data it has been given by comparing its own predictions to the DM’s predictions.
Full details of how this loss function is calculated are provided in Appendix E.
R ESULTS
Experimental Setup .
To evaluate our models, we measure three quantities: classiﬁcation error, disparate impact, and deferral rate.
We train an independent model to simulate predictions made by an external DM.
This DM is trained on a version of the dataset with extra attributes, simulating the extra knowledge/accuracy that the DM may have.
However, the DM is not trained to be fair.
When our model outputs IDK we take the output of the DM instead (see Figure 1).
Datasets and Experiment Details .
We show results on two datasets: the COMPAS dataset (Kirchner et al., 2016), where we predict a defendant’s recidivism (committing a crime while on bail) without discriminating by race, and the Heritage Health dataset, where we predict a patient’s
Charlson Index (a comorbidity indicator related to likelihood of death) without discriminating by age.
For COMPAS, we give the DM the ground truth for a defendant’s violent recidivism; for Health, we give the DM the patient’s primary condition group.
Appendix C contains additional details on both datasets.
We trained all models using a one-hidden-layer fully connected neural network with a logistic or ordinal regression on the output, where appropriate.
We used 5 sigmoid hidden units for COMPAS and 20 sigmoid hidden units for Health.
We used ADAM (Kingma & Ba, 2014) for gradient descent.
We split the training data into 80% training, 20% validation, and stopped training afterconsecutive epochs without achieving a new minimum loss on the validation set.
In the ordinal regression model, we trained with soft thresholds since we needed the model to be differentiable end to end.
In the post-hoc model, we searched threshold space in a manner which did not require differentiability, so we used hard thresholds.
This is equivalent to an ordinal regression which produces one-hot vectors i.e. P, I, N ∈ { 0 , 1 } .
See Appendices D and E for additional details on both of these cases.
Displaying Results .
Each model contains hyperparameters, such as the coefﬁcients ( α, γ ) for training and/or post-hoc optimization.
We show the results of several models, with various hyperparameter settings, to illustrate how they mediate the tradeoff of accuracy and fairness.
Each plotted point is a median of 5 runs at a given hyperparameter setting.
We only show points on the Pareto front of [Figure] [Figure] the results, i.e., those for which no other point had both better error and DI.
Finally, all results are calculated on a held-out test set.
7. L EARNING TO P UNT AND D EFER
In Figure 3, we compare punting models to binary models, with and without fairness regularization.
These IDK models have not learned to defer, i.e. they did not receive access to the DM scores during training (see Fig. 4).
The results show that, on both datasets, the IDK models achieve a stronger combination of fairness and accuracy than the binary models.
Graphically, we observe this by noting that the line of points representing IDK model results are closer to the lower left hand corner of the plot than the line of points representing binary model results.
Some of this improvement is driven by the extra accuracy in the DM.
However, we note that the model-DM combination achieves a more effective accuracy-fairness tradeoff than any of the three baselines: the accurate but unfair DM; the fair but inaccurate binary model with DI regularization; and the unfair and inaccurate unregularized
[Figure]
binary model.
Learning to punt can therefore be a valuable tool for anyone who designs or oversees a many-part system - a simple ﬁrst stage capable of expressing uncertainty can improve the fairness of a more accurate DM.
Figure 4 demonstrates a clear improvement over the punting models (DM-unaware, Sec. 5).
If we have access to examples of past DM behavior, learning to defer provides an effective way to improve the fairness of the entire system .
For insight here, we can inspect the different roles IDK plays in their respective loss functions.
In the DM-unaware IDK model, punting is penalized at a constant rate, determined by γ .
However, in the DM-aware model, deferring penalized in a way which is dependent on the output of the DM on that example.
We can consider the unaware model to be optimizing the expected DM-aware loss function for a DM with constant expected loss on each examples, such as an oracle (see Appendix A).
Then, we can see any improvement by the DM-aware model as effective identiﬁcation of the examples on which the expected loss of the DM is unusually high; in other words, identifying the inconsistencies or biases of the DM.
7. R ESULTS : D EFERRING TO A B IASED DM
Punt Defer Punt Defer Model 0.
P e r c e n t a g e o f I D K s p r
e
d
i c t e d
A=0A=1A=0,Y=0A=0,Y=1A=1,Y=0A=1,Y=
One advantage of deferring is that it can account for speciﬁc characteristics of a DM.
To test this, we considered the case of a DM which is extremely biased (Fig. 5).
We ﬁnd that the advantage of a deferring model holds in this case, as it compensates for the DM’s extreme bias.
We can further analyze where the model chooses to defer.
Recall that the DM is given extra information; in this case the violent recidivism of the defendant (true for about 7% of the dataset), which is difﬁcult to predict from the other attributes.
Fig.
6 compares the IDKs predicted by a punting model and a deferring model - split by group (race) on the left, and by group and violent recidivism on the right.
Both models achieved roughly 27% error; the deferring model had 2% DI and the punting model had 4%.
On the left, we see that the deferring model says IDK adjust this way; the deferring model can, since it receives noisy access to this information through the DM scores in training.
C ONCLUSION
In this work, we propose the idea of learning to defer.
We propose a model which learns to defer fairly, and show that these models can better navigate the accuracy-fairness tradeoff.
We also consider deferring models as one part of a decision pipeline.
To this end, we provide a framework for evaluating deferring models by incorporating other decision makers’ output into learning.
We give an algorithm for learning to defer in the context of a larger system, and show how to train a deferring model to optimize the performance of the pipeline as a whole.
This is a powerful, general framework, with ramiﬁcations for many complex domains where automated models interact with other decision-making agents.
A model with a low deferral rate could be used to cull a large pool of examples, with all deferrals requiring further examination.
Conversely, a model with a high deferral rate can be thought of as ﬂagging the most troublesome, incorrect, or biased decisions by a DM, with all non-deferrals requiring further investigation.
Automated models often operate within larger systems, with many moving parts.
Through deferring, we show how models can learn to predict responsibly within their surrounding systems.
Automated models often operate within larger systems, with many moving parts.
Through deferring, we show how models can learn to predict responsibly within their surrounding systems.
Building models which can defer to more capable decision makers is an essential step towards fairer, more responsible machine learning.
A C OMPARISON OF O RACLE T RAINING TO IDK DM-U NAWARE
In Section 7. we discuss that DM-unaware IDK training is similar to DM-aware training except with a training DM who treats all examples similarly, in some sense. Here we show experimental evidence.
The plots in Figure 7 compare these two models: DM-unaware and DM-aware with an oracle at training time and the standard DM at test time.
We can see that these models trade off between error rate and DI in almost an identical manner.
We can show that when for a broad class of objective functions (including classiﬁcation error and cross entropy), these are provably equivalent.
Note that this class does not include our fairness regularizer; for that we show the experimental evidence in Figure 7.
Let Y be the ground truth label, Y DM be the DM output, Y model be the model output, s be the IDK decision indicator ( s = 0 for predict, s = 1 for IDK), and Y sys be the output of the joint DM-model system.
Suppose these are all binary variables.
The standard rejection learning (DM-unaware) loss (which has no concept of Y DM ) is (Cortes et al., 2016):
[Equation]
We can describe the learning-to-defer system output as
[Equation]
If we wish to train the system output Y sys to optimize some loss function L ( Y, Y sys ) , we can simply train s and Y model to optimize L ( Y, sY DM + (1 − s ) Y model ) .
This deferring framework is strictly more expressive than the rejection learning model, as it can be used on many different objectives L , while rejection learning is mostly used with classiﬁcation accuracy.
We now show that if we take the DM to be an oracle (always outputs ground truth), learning to defer reduces to rejection learning for a broad class of objective functions, including classiﬁcation error, cross entropy, and mean squared error.
Theorem.
Let Y sys =
sY DM +(1 − s )
Y model , where s ∈ { 0 , 1 } .
Let L ( Y, Y sys )
= (cid:80) i (cid:96) ( Y i , Y sysi ) be the objective we aim to minimize, where Y i =
arg min
Y sysi (cid:96) ( Y i , Y sysi ) and (cid:96) ( Y i , Y i ) = 0 .
Then, if the DM is an oracle, the learning-to-defer and learning-to-punt objectives are equivalent.
Proof.
As in Eq. 8, the standard rejection learning objective is
[Equation]
where the ﬁrst term encourages a low loss (cid:96) for non-IDK examples and the second term penalizes
IDK at a constant rate, with γ punt ≥ 0 .
In rejection learning, (cid:96) is usually classiﬁcation error (cf.
Cortes et al.
Chow (1957)).
Note that this objective has no notion of DM output ( Y DM ).
If we include a similar γ defer penalty, the deferring loss function is
[Equation]
Now, if the DM is an oracle, then Y DM =
Y , meaning (cid:96) ( Y, Y DM ) = 0 , giving us
L defer ( Y
[Equation]
Y DM
[Equation]
Y model
[Equation]
(12) if we set γ defer = γ punt .
[Equation]
B R ESULTS : B INARY C LASSIFICATION WITH F AIR R EGULARIZATION
The results in Figures 8 and 9 roughly replicate the results from (Bechavod & Ligett, 2017), who also test on the COMPAS dataset.
Their results are slightly different for two reasons: 1) we use a 1-layer NN and they use logistic regression; and 2) our training/test splits are different from theirswe have more examples in our training set.
However, the main takeaway is similar: regularization with a disparate impact term is a good way to reduce DI without making too many more errors. [Figure] [Figure]
C D ATASET D ETAILS
We show results on two datasets.
The ﬁrst is the COMPAS recidivism dataset, made available by
ProPublica (Kirchner et al., 2016) 1 .
This dataset concerns recidivism: whether or not a criminal defendant will commit a crime while on bail.
The goal is to predict whether or not the person will recidivate, and the sensitive variable is race (split into black and non-black).
We used information about counts of prior charges, charge degree, sex, age, and charge type (e.g., robbery, drug possession).
We provide one extra bit of information to our DM - whether or not the defendant violently recidivated.
This clearly delineates between two groups in the data - one where the DM knows the correct answer (those who violently recidivated) and one where the DM has no extra information (those who did not recidivate, and those who recidivated non-violently).
This simulates a real-world scenario where a DM, unbeknownst to the model, may have extra information on a subset of the data.
The simulated DM had a 24% error rate, better than the baseline model’s 29% error rate.
We split the dataset into 7718 training examples and 3309 test examples.
The second dataset is the Heritage Health dataset 2 .
This dataset concerns health and hospitalization, particularly with respect to insurance.
For this dataset, we chose the goal of predicting the Charlson
Index, a comorbidity indicator, related to someone’s chances of death in the next several years.
We binarize the Charlson Index of a patient as 0/greater than 0.
We take the sensitive variable to be age and binarize by over/under 70 years old.
This dataset contains information on sex, age, lab test, prescription, and claim details.
The extra information available to the DM is the primary condition group of the patient (given in the form of a code e.g., ’SEIZURE’, ’STROKE’, ’PNEUM’).
Again, this simulates the situation where a DM may have extra information on the patient’s health that the algorithm does not have access to.
The simulated DM had a 16% error rate, better than the baseline model’s 21% error rate.
We split the dataset into 46769 training examples and 20044 test examples.
D D ETAILS ON O PTIMIZATION : H ARD T HRESHOLDS
We now explain the post-hoc threshold optimization search procedure we used.
In theory, any procedure can work.
Since it is a very small space (one dimension per threshold = 4 dimensions), we used a random search.
We sampled 1000 combinations of thresholds, picked the thresholds which minimized the loss on one half of the test set, and evaluated these thresholds on the other half of the test set.
We do this for several values of α, γ in thresholding, as well as several values of α for the original binary model.
We did not sample thresholds from the [0, 1] interval uniformly.
Rather we used the following procedure.
We sampled our lower thresholds from the scores in the training set which were below 0.5, and our upper thresholds from the scores in the training set which were above 0.5.
Our sampling scheme was guided by two principles: this forced 0.5 to always be in the IDK region; and this allowed us to sample more thresholds where the scores were more dense.
If only choosing one threshold per class, we sampled from the entire training set distribution, without dividing into above 0.5 and below 0.5.
This random search was signiﬁcantly faster than grid search, and no less effective.
It was also faster and more effective than gradient-based optimization methods for thresholds - the loss landscape seemed to have many local minima.
E D ETAILS ON T RAINING WITH E XPECTED L OSS : S OFT T HRESHOLDS
We go into more detail regarding the regularization term for expected disparate impact in Equation 7.
When using soft thresholds, it is not trivial to calculate the expected disparate impact regularizer:
[Equation]
(13) due to the difﬁculties involved in taking the expected value of an absolute value.
We instead chose to calculate a version of the regularizer with squared underlying terms:
[Equation]
Then, we can expand ˆ Y i as
[Equation]
where ˜ Y i ∈ [0 , 1] and S ( x i ) ∈
[0 , 1] are the DM and machine predictions respectively.
For brevity we will not show the rest of the calculation, but with some algebra we can obtain a closed form expression for DI soft ( Y, A, ˆ Y ) in terms of Y, A, ˜ Y and S .
F R ESULTS : L EARNING TO D EFER , BY D EFERRAL R ATE
Models which rarely defer behave very differently from those which frequently defer.
In Figure 10, we break down the results from Section 7.1 by deferral (or punting) rate.
First, we note that even for models with similar deferral rates, we see a similar fairness/accuracy win for the DM-aware models.
Next, we can look separately at the low and high deferral rate models.
We note that the beneﬁt of DM-aware training is much larger for high deferral rate models.
This suggests that the largest beneﬁt of learning to defer comes from a win in fairness, rather than accuracy.
[Figure]