Diagnosing failures of fairness transfer across distribution shift in real-world medical settings
Abstract
Diagnosing and mitigating changes in model fairness under distribution shift is animportant component of the safe deployment of machine learning in healthcaresettings.
Importantly, the success of any mitigation strategy strongly depends onthe structure of the shift.
Despite this, there has been little discussion of howto empirically assess the structure of a distribution shift that one is encounteringin practice.
In this work, we adopt a causal framing to motivate conditionalindependence tests as a key tool for characterizing distribution shifts.
Using ourapproach in two medical applications, we show that this knowledge can helpdiagnose failures of fairness transfer, including cases where real-world shifts aremore complex than is often assumed in the literature.
Based on these results, wediscuss potential remedies at each step of the machine learning pipeline
Introduction
As progress is made to integrate machine learning (ML) systems in real-world applications, oneconcern is how the ML model’s behaviour might be affected by changes in the data distribution.
While extensive work has investigated the effect of distribution shift on ML model performance[see 48, 77, 21, 74, for reviews], recent work has also highlighted that fairness properties that aresatisﬁed on the data used to develop a model might not always hold if the data distribution changes[34, 31].
In high-risk domains like healthcare, this can translate to ‘if a model satisﬁes fairness criteriawhen trained in “Hospital A”, it may not satisfy them when tested in “Hospital B”’.
Fueled by thisobservation, a new ﬁeld of research is emerging to understand the relationship between ML fairnessand robustness to distribution shift [e.g. 61, 1, 72, 16, 85, 65] and to provide mitigation techniquesfor obtaining robustly fair ML models [e.g. 64, 67, 66, 86, 15].
Importantly, the success of robustly fair machine learning depends strongly on the nature of thedistribution shifts that the system will encounter in practice.
This point has been made elegantlyin existing work as a motivation to use causal structure to provide fairness guarantees for speciﬁeddistribution shifts [64, 67].
However, to the best of our knowledge, few methods can diagnose thenature of the shift and guide towards an appropriate mitigation strategy [66].
In this work, we design statistical tests that can, based on a simpliﬁed causal graph of an applicationand a small target dataset, assess the structure of the distribution shift encountered by the systemand identify aspects of the data that hinder the transfer of fairness properties.
Using applications indermatology and Electronic Health Records (EHR), we show that this analysis is crucial for selectingan appropriate mitigation strategy.
Further, our work also reveals that clinically plausible shifts areoften more complex than can be handled by current mitigation techniques, highlighting the need foradditional research into a broader set of remedies across the entire ML pipeline.
Background
Notation We consider problems where we are given variables X , A , Y , where X is a set of features,
A is one or more sensitive attributes, and Y is an outcome or label to be predicted.
Our goal is tobuild a classiﬁcation or regression model f
( X (cid:48) ) , where, depending on the context, X (cid:48) :
=
( X, A )or X (cid:48) :
= X .
We are concerned with the fairness properties of f , both on the training distributionand in one or more deployed environments.
Depending on the setting, we focus on several statisticaldeﬁnitions of fairness [5]: gaps in subgroup accuracy, demographic parity, and equalized odds.
See
Sec.A.1 for their formulation.
Formally, we represent distribution shifts using the so-called Joint Causal Inference (JCI) frameworkof Mooij et al.
We represent causal relationships between elements of A , X , Y in a causal
Bayesian network (CBN, see Sec.
A.2 for an introduction).
In a CBN, a node U i with an arrowinto U j is called a direct cause of U j .
We call all direct causes of a node U i its causal parents,and denote them pa
( U i ) .
To represent shifts, we augment the CBN with an environment variable
S , where the event S = 0 indicates data from the source environment and S = 1 denotes datafrom the target environment, so that source data are distributed according to P ( A, X, Y | S = 0)and
the target data to P ( A, X, Y | S = 1) .
This is called the joint causal graph .
In the jointcausal graph, arrows from S denote changes to statistical relationships that are induced by the shift.
Speciﬁcally, an arrow from S to some variable U indicates that the conditional distribution of thatvariable given its causal parents could change between training and deployment environments, whilethe absence of such an arrow would indicate that the conditional distribution remains the same, i.e.
P ( U | pa ( U ) , S = 0) =
P ( U | pa ( U ) , S = 1) .
Components of Distribution Shifts In JCI formalism, the components of a distribution shift arerepresented by direct arrows from the shift variable S to variables A , Y , and X .
Here, we brieﬂysummarize how these arrows can be interpreted.
S
→ A arrows represent demographic shifts ,implying P ( A | S = s ) changes for different values of s .
An example of such a shift is a differencein the age distribution of patients in S = 0 versus S = 1 .
S
→ X arrows represent covariate shifts 1 ,where the conditional distribution of input features P ( X | pa ( X ) , S = s ) changes for differentvalues of s .
For example such a shift could result form using different cameras or views to acquire animage in S = 0 and S
= 1 (c.f., acquisition shift in Castro and Glocker [7]).
Finally, S → Y arrowsrepresent label shifts 2 , where the conditional distribution of labels P ( Y | pa ( Y ) , S = s ) changesfor different values of s .
A label shift can occur through different disease prevalences, or differentstrategies for obtaining labels, across S = 0 and S
= 1 (c.f., prevalence shift and annotation shift,respectively, in Castro and Glocker [7]).
In real-world applications, distribution shifts often include multiple components, where there aremultiple arrows from S to { A, X, Y } .
We call such shifts compound .
Compound shifts often occurwhen there is an unobserved variable that affects multiple variables whose distribution changesbetween source and target.
An example is that of a new device on the market [23].
Such a devicecould create qualitatively different images X , but it could also change the observed patient populationby being made more available to certain subsets of demographic groups that tend to be more or lesshealthy.
In this case, the shift induced by introducing the device simultaneously affects A, X and Y .
Shift Structure and Fairness Transfer
The transfer of fairness depends on the causal structure ofthe application and the structure of the shift [64, 65].
In particular, implications for fairness transfercan be directly read off of conditional independences with respect to the shift S in the joint causalgraph.
For example, for classiﬁers that take a set of variables { V } as input, the following metrics areknown to be invariant across the shift under the following conditional independence conditions:
• Classiﬁcation error ( P ( Y (cid:54) =
f ( V ) | V , S ) ) is invariant if Y ⊥⊥ S | { V } [39, 64].
• Demographic parity gap ( max a ∈A E [ f ( V ) | S, A
= a ] − min a ∈A E [ f ( V ) | S, A = a ] ) isinvariant if { V } ⊥⊥ S |
A [64].
• Equality of odds gap ( max a,y ∈AY [ f ( V ) | S, A
= a, Y
= y ] − min a,y ∈AY [ f ( V ) | S, A =a, Y
= y ] ) is invariant if { V } ⊥⊥ S | Y, A [64].
Algorithm 1 (Conditional) independence testing assessing the nature of shift S on a single variable
U ∈ G .
U represents the feature values or its summary if high-dimensional.
(cid:12) is the Hadamard product.
Require: source dataset D , target dataset D (cid:48) , a joint causal graph G
Split D into D w , D t ; Split D (cid:48) into D (cid:48) w , D (cid:48) t for 1: n bootstrap do
Sample V from D n 0 × l t and V (cid:48) from D (cid:48) n 1 × l t
Set w s = 1 n s for s ∈ { 0 , 1 } .
if pa ( U ) not ∅ then (cid:46) Compute balancing weights
Sample dataset Q = {
U )
S
D w
U )
S
D w
Build predictor P ( S | pa ( U )) classifying Q from Q w s end if end for
Compute return value (cid:46)
The p U ⊥⊥ S | U ) In JCI formalism, the above conditions imply that fairness transfer requires certain arrows from S to{ A, X, Y } to be missing.
The nature of the shift also strongly inﬂuences which mitigation strategiesare applicable.
We discuss implications for mitigation strategies in more detail in Sec.5.
Empirical tests for assessing shift structure
Given the central role of distribution shift structure in fairness transfer, our main methodologicalproposal is to investigate the structure of shifts empirically when fairness fails to transfer.
In thissection, we outline a general strategy for testing which components of a shift are present, given datafrom source and target distributions, and a BCN for the application.
Approach.
For each variable U in the graph (i.e. each A , X and Y ), we assess whether there is adirect effect of the environment S on U .
We test the null hypothesis H 0 : P ( U | pa ( U ) , S = 0) =
P ( U | pa ( U ) , S = 1) .
Rejecting this hypothesis implies an arrow S → U in the joint causal graph.
If the marginal distribution of U differs across source and target distributions, this comparison can bechallenging.
To isolate the direct effect of S , we compute weights that balance the causal parentspa ( U ) across environments [55, 28].
We then test the hypothesis that the reweighted distributionsof U match across the environments S .
For simplicity, we test whether the reweighted means areequal, and in cases where the dimensionality of U is high, we conduct this test on functions of U thatreduce the dimensionality, following strategies in Rabanser et al.
This procedure is describedin Algorithm 1 and detailed in Sec. A.3.
Validation.
We conduct three validation experiments for our testing procedure using data from ourdermatology application in Sec.
A.3.3.
We ﬁrst assess Type I error in an experiment where we testrandom splits of the same data against each other.
Our test displays a false positive rate similar to ourthreshold of 5% for hypothesis testing.
We however note that the variance of this result increases withsmaller numbers of samples.
We then conﬁrm non-trivial power in an experiment where we introducea shift by subsampling younger patients with a particular skin condition in the source dataset (a shifton A and Y ).
Our testing approach correctly identiﬁes the dimensions of A and Y that were affectedby S due to the engineered shift. We replicate this test for a shift in Y only, with the same result.
Distribution shifts in real-world healthcare applications
Here, we present two case studies from the healthcare domain in dermatology and in clinical riskprediction using Electronic Health Records in which fairness does not transfer (Fig. 1).
As per [32],3 A rejection of these simpliﬁed
[Figure]
we consider the setting of ‘dataset shift’, whereby a model is developed on the source data and testedon the target data 4 , which is a common setting in medical applications [78, 84, 32].
4.1 Predicting skin conditions in dermatology
In this task, we predict common skin conditions (26 conditions plus an additional ‘other’ category tocapture the long tail), based on one or multiple images of the pathology of interest and additionalmetadata [38].
The data used as source is a subset of the data described in Liu et al.
Brieﬂy, itconsists of de-identiﬁed retrospective adult cases from a teledermatology service serving 17 sitesfrom 2 states in the United States.
Each case contained 1-6 clinical photographs of the affectedskin areas as well as patient demographic information and medical history (see Table S1 of [38]).
We split the source data such that it comprises 12,024 cases for training, 1,925 for validation andhyper-parameter tuning and 1,924 for testing.
We consider how the fairness properties of a model trained in this setting transfer to a target dataset,unobserved at model development time, consisting of 1,843 labelled teledermatology cases from
Colombia.
In the Supplement, we consider transfer to a second target from skin cancer clinics in
Australia, as well as the properties of a model jointly trained on the source and the second target, anddeployed on the ﬁrst target data.
The model we consider is a ResNet architecture, as in [38, 57].
We assess model performance and fairness across 10 replicates of the model on the source and onthe target (Fig. 1).
On the source data, the model performs similarly across all subgroups and couldreasonably be assessed as ‘fair’.
However, the maximum gap between age groups increases starklyon the target (from 1 .
05% to 12 .
To understand the failure of fairness transfer in this context, we investigate the structure of the shiftby referring to Algorithm 1.
We begin by drawing a simpliﬁed causal graph G of the dermatologytask at hand (Fig.2(a)).
We consider this an anti-causal learning task [60], in which the skin conditionlabel Y is a cause of the input image X , and consider demographic information A to be a causalparent of both X and Y .
We omit variables that are unobserved in either the source or target data.
We now test whether the shift S has direct effects on A , Y , or X .
Overall, we ﬁnd that the shift iscompound, and affects all three aspects of the data distribution.
We begin with A , which comprises [Figure] age and sex.
Because A has no causal parents in our graph (besides, potentially, S ) we directly assessdifferences in age and sex across environments using classical tests.
Age has a different distributionacross the two datasets (see Fig. 2(b)), with the population in the source data being typically younger(median: 40 years old, 25 % quantile: 27, 75 % quantile: 54) than in the target data (median: 44 yearsold, 25 % quantile: 30, 75 % quantile: 59).
Sex distributions are well matched across the source and target.
These results suggest a direct effect of S on A , and we add this relationship to the causalgraph (Fig. 2(a), in purple).
We then assess whether S directly affects the labels Y .
In our graph, A is a causal parent of Y , sofollowing our testing strategy in Algorithm 1, we test whether P ( Y | A, S = 0)
=
P ( Y | A, S = 1) .(Recall
that conditioning on A isolates the direct effect of S on Y .)
Using 27 univariate tests (one percondition, [54]), we obtain signiﬁcant differences between P ( Y | A, S = 0) and P ( Y | A, S =
1)for 3 conditions ( p < 0 .
05 , Bonferroni corrected).
Fig.2(c) illustrates the label shift in a speciﬁc ageand sex subgroup (here females aged 65+, 409 cases in the source, 128 in the target data).
We seethat the source data includes more cases of ‘eczema’ and ‘psoriasis’ while conditions like ‘actinickeratosis’ and ‘seborrheic dermatitis’ are more represented in the target.
These results suggest thatthe environment also directly affects the labels (orange link in Fig.2(a)).
Finally, we consider whether the features of the images themselves (designated by X ) are directlyaffected by S .
Conditioning on causal parents of X , we test whether P ( X | Y, A, S = 0) =
P ( X |
Y, A, S = 1) .
We summarize X by training a model f ( X ) →
Y on the source, a strategy proposed in
Rabanser et al.
Our weighted tests suggest a signiﬁcant difference between the two distributions( p < 0 .
05 corrected on 1 dimension).
Fig.2(d)
illustrates this difference by computing the number ofimages per case in the group of older females considered above with cases labelled as ‘SK/ISK’ (source median: 3, n = 61 , target median: 5, n = 18 ).
This result suggests the existence of the directpath S
→ X
and we add this relationship to the graph (green link in Fig.2(a)).
Based on these results, we observe that all aspects of the data are affected by the environment.
Thus,none of the conditional independence conditions for fairness transfer hold in this setting.
However,we also see that some conditions might display invariance once controlled for demographics, whichsuggests possible mitigations that we explore in Section 5.
The Supplement includes experimentaldetails and further results on the second target and the joint training.
4.2 Clinical risk prediction from Electronic Health Records
In this application, we predict the clinical outcomes of patients based on EHRs.
EHRs record thetime series of interactions between a patient and the clinical system (e.g. medication, labs, vitals, ...).
Distribution shift is a major concern for systems that consume EHR data, which can be heterogeneousacross centers [79, 65], and even within a single system [45].
As an example, over half of hospitals inthe US with intensive care units (ICUs) only have a single unit where all critically ill patients (medical,surgical, cardiac, etc.) are admitted 7 .
Risk scores, such as the Acute Physiology and Chronic Health
Evaluation II (APACHE II) score are used to evaluate all critically ill patients, but do not performequally across clinical subgroups, e.g. underperforming on patients with cardiac disease [53].
Itwould be desirable to avoid such issues with risk scores constructed through machine learning.
Weuse the open access, de-identiﬁed Medical Information Mart for Intensive Care III (MIMIC-III)dataset [29], which consists of data from admissions to ICUs at the Beth Israel Deaconess Medical
Center between 2001 and 2012.
This clinical system has the beneﬁt of having separate specialized
ICUs and allows to assess the generalizability of risk scores estimated in one clinical population toanother.
Based on clinical input, we consider the Medical ICU (MICU), Surgical ICU (SICU) and
Trauma Surgical ICU (TSICU) to be generalist ICUs (source data); whereas the Cardiac Surgery
Recovery Unit (CSRU) and Coronary Care Unit (CCU) are specialized ICUs (target data).
This splitleads to 17,641 patients included in the source dataset and 10,442 patients in the target dataset, afterselecting adult patients with a length of stay of minimum one day and with a recorded care unit. Ourgoal is to obtain a robustly fair model that predicts prolonged ICU stay (i.e. length of stay > 3 days,as in [75, 52]) using data from the ﬁrst 24 hours of ﬁrst ICU admission and a recurrent architecture[69, 70, 58]. The model is trained on 80% of the source data, tuned using a separate split of 10% andtested on the remaining 10% . See the Supplement for further details.
Noteworthy, the model does nothave access to the ‘reason for visit’.
Figure 1(b) shows that the fairness properties of this model, as measured by gaps in subgroupperformance, demographic parity (DP) and equalized odds (EO), do not transfer across ICU types.
The model displays unfairness w.r.t.
age in both datasets (maximum gap ∼ 7% , similar DP and EOscores).
However, the pattern of unfairness is different in the two datasets, with e.g. the 45-64 yearsold subgroup being under-served in the source but over-served in the target.
The performance gapbetween sex groups increases from 1% in the source (DP: 0 .
002 ± 0 . 002 ) to 2 .
7% (DP: 0 .
016 ± 0 . 003 )on the target.
To understand this failure of fairness transfer, we examine the structure of the shift.
First, we drawa causal graph for the variables in this problem, building on the work in Singh et al.
Here,we incorporate demographics A as age and self-reported sex, comorbidities M as previous medicalhistory deﬁned by ICD-9 codes [9, 17, 27, 35], observed features X as all labs and vitals, and Y asour prolonged length of stay label.
As a more comprehensive feature set leads to better predictiveperformance [58], we use all 59,351 features at our disposal and add ‘treatments’ T (e.g. medications)to the graph (Fig. 3(a)).
In our graph, S represents the ICU unit type that a patient is admitted to.
We now test whether the shift S has direct effects on the variables in this problem using our proposedapproach.
Beginning with A , which has no causal parents, we test whether P ( A | S = 0) =
P ( A | S = 1) .
We estimate the distribution of age and sex in each dataset (Fig. 3(b)).
We observe that thepopulation in the source is typically younger than the population in the target (t-test: p < 0 . 0001 ).
Inaddition, the proportion of males in the source and target differ (source: 50%, target: 65%).
Therefore, S → A .
To assess the shift’s effect on comorbidities M , we refer to Elixhauser and Vanwalraven scores[20, 71] computed on the source and target data [30] 8 .
Figure 3(c) displays the distribution of the30 comorbidities for older males ( n = 3 , 951 in source and n = 3 , 588 in target).
As expected, [Figure] even after adjusting for the causal parent A , patients in the target data have a higher prevalence ofcomorbidities related to the peripheral vascular system than in the source data and M (cid:54)⊥⊥ S | A for 5comorbidities (weighted tests: p < 0 .
05 , Bonferroni corrected).
We repeat the analysis to assess the direct effect of S on the treatments ( P ( T | A, M, S =0) =
P ( T | A, M, S = 1) : p < 0 . 05 ), on the labs and vitals ( P ( X | A, M, T, S = 0) =
P ( X | A, M, T, S = 1) : p > 0 . 05 ) and on the label ( P ( Y | A, M, T, S = 0) =
P ( Y | A, M, T, S =1) : p > 0 . 05 ) in the Supplement.
Figure 3(d) displays the prevalence of prolonged length of stay forolder males with a solid tumor comorbidity and receiving vasopressors/inotropes ( n = 55 in sourceand n = 16 in target).
Our results suggest that the shift is compound, i.e. the ICU unit a patient is admitted to changesthe relationships between multiple variables in our causal graph.
We note that this conclusion isnot surprising clinically, given that the environment S is correlated with the reason for their ICUadmission, and that this reason for admission (unobserved) is a main driver for all variables.
As inthe dermatology case, the compound shift leads to multiple paths for S and A to affect the covariates
M and X and the label Y , explaining the failure of fairness transfer.
Mitigation and related work
A variety of fairness, robustness and robustly fair mitigation strategies have been proposed.
Wecontextualize a number of methods in terms of anti-causal and causal learning tasks [60] in Sec.
B,and highlight how understanding the nature of the distribution shift can guide mitigation.
Fairness mitigation.
Pre-, in- and post-processing techniques for mitigating unfairness have beenproposed [see 5, 46, for overviews].
Fairness metrics and mitigation techniques currently usedin real-world applications are typically evaluated on a single learning task or data distribution [e.g.87, 2, 76, 52, 11].
More similar to our settings, the work by Creager et al.
[15] addresses fairness indynamical systems, although only for low-dimensionality variables.
Based on our causal framing,applying fairness mitigations in the source domain could potentially aid in fairness transfer by cuttingsome edges in the underlying CBN [72].
However, fairness mitigation on the source only leads toprovable fairness transfer in speciﬁc settings, i.e. mostly when the shift is demographic (Sec.B).
Based on this analysis, performing fairness mitigation in the presence of a compound shift would failto provide a fair result on the target.
Considering the applications above, a reasonable option wouldbe to perform fairness mitigation for the EHR task based on age.
We perform post-processing of thepredictions to enforce either demographic parity or equalized odds across age subgroup using themethod in Alabdulmohsin and Luˇci´c [3].
This results in a decrease in demographic parity to value10 − 5 in the source, but to (slightly) increase to value 0 .
078 ± 0 .
025 in the target data.
Similarly,equalized odds decrease to value 10 − 5 and the gap in model accuracy between groups is reduced from7% to 0.8% (Fig. 4) in the source, but increases from 0 .
05 ± 0 .
008 to 0 .
16 ± 0 .
04 on the target data,and the gap between age groups increases to 11.4%.
We therefore observe that mitigating for eitherdemographic parity or equalized odds on the source leads to no improvement, or even worsening ofthe fairness properties on the target, as predicted by our analysis.
Robustness mitigation: Relations between robustness guarantees and individual and group fairnesshave been studied in [82, 44, 80, 81] and [72] respectively.
[1, 16, 85] have investigated the relationship between fairness and distribution shift experimentally by studying how robustness mitigationstrategies affect fairness metrics.
These works do not provide provable robustly fair models.
Robustly fair mitigation.
Work explicitly investigating fairness properties under distribution shiftcan mostly be divided into methods that provide a priori guarantees or bounds [e.g. 61, 64, 67] andmethods that adapt to the new distribution [e.g. 13, 47, 66, 86, 31, 18, 68].
Recent work by Singh et al.
[64] suggests a straightforward mitigation with guarantees, based onselecting features that satisfy the fairness transfer criteria outlined in Section 2.
However, thisapproach is limited under compound shifts with complex data.
Most notably, it cannot be usedin cases where there is a label shift ( S → Y ).
However, even when there is no label shift, undercompound shifts the criteria for feature selection can yield a trivial predictor where no observedvariables can be used.
This is the case in our EHR example under the joint causal graph in Fig. 3(a).
We discuss the applicability of [64] across different causal scenarios in Sec.
B.3.
In addition, featureselection approaches are more suited for semantic features and, while it is technically feasible, wouldnot lead to insightful results in non-semantic features such as pixels.
In computer vision, maskinghas been used previously as an approximation to feature selection to constrain the learning towardsspeciﬁc areas of the image [56, 63].
While this technique has shown some promise with chest x-rays[73], it is impractical due to the additional requirement of a mask for all training images.
In addition,it increases the ratio of signal of interest compared to undesirable signal, but does not exclude suchsignals and comes with no guarantee.
Another line of work considers adapting transfer learning and meta-learning algorithms to considerfairness constraints and therefore ensure generalization to new tasks also in terms of fairness [13,47, 66, 86].
Speciﬁcally, Coston et al.
[13] modify the standard approach to transfer learning undercovariate shift by adding a constraint to the weight learning objective that enforces closeness betweenall pairs of protected groups.
Schumann et al.
[61] provide bounds on fairness transfer by addingadversarial heads on both the in-source attribute A and the environment S .
When we cast thiswork into our causal framework, we notice that it would provide a robustly fair model under anexclusive covariate shift in an anti-causal learning task.
Slack et al.
[66] propose meta-learning with aregularizer that imposes demographic parity or equalized odds to improve the transfer of fairness.
Themethod requires a small set of labelled target data (K-shot learning) and multiple tasks, and comeswith no guarantees.
Work on ﬁne-tuning and transfer learning [47, 86] leverages task similarity tolearn fair representations that provably generalize to unseen tasks, similarly requiring a small set of labelled target data.
Importantly, these methods generate one tuned model per environment, which isnot suitable for all real-world deployments of ML.
It is also unclear whether these techniques couldhandle more complex tasks, e.g. predictions from sparse multivariate timeseries as in the case of
EHR.
Shift detection.
Many methods exist to detect covariate shift [e.g. 40, 8] or label shift [37, 24].
Theyhowever often rely on the assumption that other variables are not affected by the shift.
For instance,
Lipton et al.
[37] relies on the assumption that X ⊥⊥ S | Y .
On the other hand, Slack et al.
[66]introduce a technique to identify mean shifts that could lead to unfairness.
While the method is aninteresting step towards ‘fairness transfer warnings’, this strategy does not allow to guide the selectionof mitigation.
Recently, Singh et al.
[65] investigate differences in source and target distributionsusing squared maximum mean discrepancy [26], but do not take the causal structure into account.
To differentiate direct from indirect effects of S on variables U , they perform causal inference andvisually inspect the obtained graph.
This method can only be applied to low-dimensionality datasets.
Discussion
In this work, we applied statistical tests within a joint causal framework [43] to better understandfailures of fairness transfer across distribution shift.
Our approach tests for direct effects of theenvironment on every random variable in a task.
We use this technique to characterize distributionshifts that result in two real world failures of fairness transfer in the healthcare domain.
We show thatthis knowledge can then guide the choice of an appropriate mitigation strategy, when applicable.
Our work highlights real-world challenges that prevent the development of robustly fair models.
Giventhat each task leads to different complexities in terms of causal graph, expected shifts, and fairnessrequirements, these ﬁndings support looking beyond purely algorithmic solutions.
Considering thewhole ML pipeline, we can re-interpret the remedies discussed in Chen et al.
[10] for fairness transfer:
Problem selection.
Consider focusing on tasks with lower anticipated harms or clini-
cal/policy safeguards against ML unfairness
Data collection.
Consider early data collection from anticipated deployment environments
in which distribution shifts are suspected.
This would allow to perform (conditional) independence tests to guide further mitigation.
Outcome deﬁnition.
Consider intermediate outcomes for which unfairness under distribu- tion shift might be less consequential (e.g. image segmentation compared to diagnosis).
Algorithm development.
Based on (2), some strategies can be used if their assumptions are
satisﬁed.
In addition, in the present work we used a causal framework.
However, techniques
that are not easily cast into this framework exist.
For instance, manual feature engineering
has been used to improve the robustness of EHR models to distribution shift [45].
While this
work has not evaluated the fairness of the model in the source and target distributions,
we
believe that feature engineering might be a promising avenue to enforce speciﬁc inductive
biases during learning while reducing the risks of distribution shifts.
We however note that
our work does not assess the impact of model architecture or different training strategies,
which might lead to compounded bias and/or poor generalization.
Post-deployment.
Further safeguards may be envisaged to safely deploy ML systems, e
g.with a prospective observational (non-interventional) integration [23].
Transfer learning
or retraining with site-speciﬁc updates might then be considered as necessary prior to interventional deployment.
Finally, those deployments may be accompanied with robust post-market surveillance to continuously monitor fairness.
Limitations and future work.
The power of our diagnosis method depends on the number andquality of the samples.
Therefore, it should be perceived as displaying potentially problematic aspectsof the data, rather than assessing invariant features.
This is a common limitation in shift detection[37] and mitigation [64].
Furthermore, the data at hand usually represents a set of proxies for theunderlying variables.
Therefore, even if a shift seems to satisfy some invariance assumptions, thereare no sharp guarantees [22].
This is especially true for demographic factors: features such asethnicity, age or sex only approximate sensitive attributes.
We are also limited by which factors areobserved across all environments and cannot assess changes in unobserved variables (e.g. socialdeterminants of health).
While our work covers different types of shift, future work should investigate further sources of shiftand biases, such as sampling biases or missingness [7, 85].
In addition, extending the testing strategyto compare multiple (discrete or continuous) versions of the environment S would be valuable.
By specifying causal graphs for each problem, we make assumptions about the relationships betweenvariables.
These assumptions might not represent the true underlying data generation process.
Inaddition, the effects of A could potentially be divided into ‘fair’ and ‘unfair’ effects, as proposedin [12, 42].
For real-world applications, further assumptions were made, e.g. considering the silverstandard label in dermatology as a good approximation for the gold standard label (hence an anticausal task)
The value of these graphs is purely illustrative and should not be considered formedical applications without validation.
In terms of metrics, we have focused mostly on demographic parity and equalized odds, due totheir recent causal grounding [72].
Different metrics might, however, be affected differently, and itis possible that some metrics are more or less ‘sensitive’ to distribution shift.
Similarly, differentfairness mitigation strategies might be affected differently by distribution shift, although none providesguarantees under distribution shift.
We also note that ‘fairness’ in the healthcare domain is an activediscussion, and that equalized odds or demographic parity metrics do not consider factors such associal determinants of health or health equity.
As discussed in [52], considering human elicitedmetrics might be an avenue forward.
Similarly, ‘fairness’ might change depending on the context[59] and different metrics might be relevant in different environments.
A Methods
In this section, we provide further details on the methods, including the mathematical deﬁnitions ofour metrics and an in-depth description of our testing procedure.
A.1 Fairness metrics
Our work focuses on statistical group deﬁnitions of fairness [5].
In particular, we frame the discussionaround demographic parity in which the model’s output is statistically independent of the sensitiveattribute, i.e. f ( X (cid:48) ) ⊥⊥ A , and equalized odds in which the independence holds conditionally on theoutcome, i.e. f ( X (cid:48) )
⊥⊥ A | Y .
In the experiments, these metrics are computed for binary tasks.
Wealso compute the gap in subgroup accuracy.
For all fairness metrics, the lower, the better.
We deﬁne demographic parity for a predictor f as [19, 83, 41, 3]:
[Equation]
Equalized odds is computed in a similar fashion by conditioning on the positive and negative classes,taking the average of the discrepancies across classes.
Subgroup performance is computed as the accuracy (top-1 or top-3 in the case of dermatology,accuracy for EHR) within each subgroup.
When the sensitive attribute is non-binary, we compute themaximum difference between any pair of subgroups.
A.2 Causal framework
A Bayesian network [14, 33, 49, 50] is a directed acyclic graph (DAG)
G whose nodes X 1 , . . .
X Drepresent random variables and links express statistical dependencies among them.
Each node X d isassociated with a conditional probability distribution (CPD) p ( X d | pa ( X d )) , where pa ( X d ) denotethe parents of X d , namely the nodes with a link into X d .
The joint distribution of all nodes isgiven by the product of all CPDs, i.e. p ( X 1 , . . .
, X D | G ) =
(cid:81)
Dd =1 p ( X d | pa ( X d )) .
This function isassumed to be invariant across distributions.
A path from X i to X j is a sequence of linked nodesstarting at X i and ending at X j .
A path is called directed if the links point from preceding towardsfollowing nodes in the sequence.
A node X i is an ancestor of a node X j if there exists a directedpath from X i to X j .
In this case, X j is a descendant of X i .
We say that a set of nodes { X i , . . .
, X j }d-separates two random variables U and W if U ⊥⊥ W | { X i , . .
X j } .
A causal Bayesian networkis a Bayesian network in which a link expresses causal inﬂuence rather than statistical dependence.
In causal Bayesian networks, directed paths are called causal paths .
A.3 Assessing the causal structure of shiftsA.3.1 General Strategy
To assess the causal structure of a shift, we examine whether there is a direct effect of the shift S ona focal variable U .
This requires controlling for all other pathways by which U may depend on S .
To do this, we select a set of variables V such that { V } blocks all indirect paths from S to U .
Wethen test the following equality in conditional distributions: P ( U | V , S = 0) =
P ( U | V , S = 1)almost everywhere.
To test the equality of conditional distributions, we reduce the problem to testing whether the meansof the two marginal distributions of U are equal when the distributions of the background variables
V are adjusted to follow the same distribution.
Using observed data from environments S = 0and S = 1 , we can perform such an apples-to-apples comparison using importance weighting.
In particular, for any distribution π ( V ) ,
[Equation]
) for almost all V
[Equation]
Deﬁne the weights w 0 ( V ) :
= π ( V ) P ( V | S =0) , and w 1 ( V ) :
= π ( V ) P ( V | S =1) .
This result shows that we cantest the equality of the conditional distributions by testing the implication that the mean of weightedoutcomes is the same.
This leaves a degree of freedom for choosing the test distribution π ( V ) on which the conditional distributions will be compared.
This distribution then determines the weighting scheme that will be used.
We deﬁne π ( V ) to be a constant, i.e., uniform.
Then the weights are
[Equation]
and w
[Equation]
V
[Equation]
∝ P ( V | S = 1) − 1 .
This weighting scheme is known as inverse probability weighting or IPW [55, 28], and is a popularchoice.
We however note that other weighting schemes could be considered, e.g. overlap weights[36] or permutation weighting [4].
In practice, the likelihoods P ( V | S = s ) used to deﬁne the weights need to be learned fromdata.
Usefully, because of how the weights are normalized, the likelihoods can be replaced withclassiﬁcation scores that estimate P ( S = s | V ) , leading to estimated weights ˆ w 0 and ˆ w 1 .
The choiceof the classiﬁer can be informed by the expected relationships between { V } and S : for instance,logistic regression can be considered in the case of simple relationships, while more expressiveclassiﬁers such as gradient boosted trees can be considered in the case of non-linear relationships [4].
As discussed in Arbour et al.
[4], all classiﬁers are tuned (C for logistic regression, number of trees,tree depth and learning rate for boosted trees) in a cross-validation setup.
We then use a standard t-test to test the null hypothesis
[Equation]
To decrease the variance in the obtained t-statistics, we bootstrap the weighting and testing procedure.
The ﬁnal p-value associated with H 0 : P ( U | V , S = 0) =
P ( U | V , S = 1) is the p-value of atwo-sided Z -test on the t-statistics obtained from each bootstrap.
Observations and tips From our observations, we note that it is best to allocate the largest availablesample to estimate the likelihood ratio, compared to performing the ﬁnal statistical test.
We alsoobserved that gradient boosted trees can lead to extreme weights in some cases.
This results in anincreased variance and non-signiﬁcant results.
We therefore clipped the weights to a maximum of 10(arbitrary choice) to mitigate this effect.
A.3.2 Testing with High-Dimensional U
If U has low dimensionality, Rabanser et al.
[54] show that multiple one-dimensional tests can beused with correction for multiple comparisons (here Bonferroni).
On the other hand, if U has higherdimensionality (e.g. an image), summaries of U can be constructed and the test can be performedon these lower-dimensional summaries.
The validity of this approach follows from the fact thatif the conditional distributions of U are the same, then so will the conditional distributions of anysummary f ( U ) .
The trade-off is that the test loses all power to detect distributional differences thatare compressed out by the summary f ( U ) , or to highlight speciﬁc dimensions in U that are more orless affected by S .
To deﬁne a summary that is relevant to the problem, Lipton et al.
[37] suggestdeﬁning f ( U ) to be the output of a model the predicts some variable of interest (say the outcome Y )using U .
As reported in Rabanser et al.
[54], other summarizing techniques could be considered.
A.3.3 Sanity checks with engineered shifts
We assess the speciﬁcity and sensitivity of our testing procedure (see Algorithm 1) using engineeredshifts on the dermatology data.
To estimate Type I error, we compare the source data to itself, whichshould lead to all variables being independent of the environment.
More speciﬁcally, we select 10 , 000random samples from the dataset to compute the weights, and another 1 , 000 random samples toapply the weights on and perform the weighted test.
These sets are boostrapped 100 times (separatelyfor the source and target sets) and we assess the proportion of false positives that is obtained for eachvariable U .
We expect that we will obtain 5% of false positives, as deﬁned by our hypothesis testingthreshold.
As we do not expect any relationship between { V } and S , we use a logistic regression toestimate P ( S = s | V ) .
When comparing P ( Y | A, S = 0) to P ( Y | A, S = 1) , we observe that the accuracy of the classiﬁermirrors the chance level of 50% and that the proportion of false positives varies between 2 and 8 .
5%(Figure 5(a)).
Similarly, comparing P ( X | Y, A, S = 0) to P ( X | Y, A, S = 1) when using a modelf ( X ) → Y to summarize X leads to a proportion of false positives between 2 and 7% (Figure 5(b)).
As the number of samples used to compute the weights is an important component of our testingapproach, we repeat the process while varying the number of samples from 100 to 5,000. Figure 5(c)displays that the variance on Type I error (here across conditions) decreases when the number ofsamples to ﬁt the P ( S = s | V ) classiﬁer increases.
[Figure]
To assess sensitivity, we then engineer a shift on the data that discards samples based on their ageand condition (here Acne).
More speciﬁcally, we subsample data in order to decrease the bias foryounger age in the Acne condition, leading to an increase in age for Acne samples of approximately10 years.
Given that Acne is one of the most prevalent conditions in the dataset, we expect a signiﬁcant change on age, as well as on Y .
On the other hand, the images are not modiﬁed and hence
P ( X | Y, A, S = 0) and P ( X | Y, A, S = 1) should not display signiﬁcant changes.
We observe a signiﬁcant change in age between S = 0 and S = 1 .
There is no signiﬁcant effectof S on other demographic factors.
We do observe a direct effect of S on Y , with a strong effecton Acne as expected.
On the other hand, no tests display signiﬁcant differences when comparing
P ( X | Y, A, S = 0) with P ( X | Y, A, S = 1) .
This latter result was obtained with both a logisticregression and gradient boosted trees (given the more complex relationship between age and Acne).
Therefore, the testing approaches produces a causal graph that mirrors the graph used to engineer theshift, and our procedure is considered faithful.
Finally, we randomly shufﬂed the labels y k across the 27 conditions, for each case independently.
At the population level, this results in a change in P ( Y ) .
As expected, there was no effect of S on
A .
For Y , we observed signiﬁcant differences in P ( Y | A, S = 0) and P ( Y | A, S = 1) for 19conditions out of 27 (Bonferroni corrected).
Thanks to the correction, and given that the images werenot modiﬁed, there are no signiﬁcant differences between P ( X | Y, A, S = 0) and P ( X | Y, A, S = 1) .
Overall, our tests are sensitive enough to detect the presence of changes on the affected dimensions,while not displaying an excessive amount of false positives.
B Causal framing of related works
In this section, we review how various transferable fairness approaches that have been proposedin the literature interact with different causal structures of domain shift depicted in Figures 6 and7.
We assume the same sensitive attributes A in the source and target environments, and considerdifferent relationships between the environment and other variables.
As in [7, 72], we split theanalysis into causal or anti-causal prediction tasks [60].
A prediction task is causal if the features Xare causes of the outcome Y , and anti-causal if the outcome Y is a cause of the features X .
Manyrisk prediction problems are causal, where risk factors may cause an adverse outcome, while manydiagnostic problems are anti-causal, where the underlying disease may cause symptoms [7].
A takeaway from this review is that most methods are tailored to particular restricted shift structures,and their useful properties can break down under compound shifts.
B.1 Anti-causal prediction tasks
First, we consider anti-causal prediction tasks, in which the outcome Y is a direct cause of thefeatures X , and that the sensitive attribute A is a direct cause of X and Y (the reality will likely bemore complex and involve other variables in the paths (observed or not)).
The assumption that A is acause of X and Y is a reasonable approximation in healthcare settings: e.g. on average, skin imagesfrom men are more hairy than those of women, breasts are visible in chest x-rays, and comorbiditiesare different across sexes and age ranges.
Sensitive attributes can also be related to the label, asprevalence can vary across subgroups (e.g. baldness is more prevalent in older patients) and someconditions might present differently across attributes (e.g. a heart attack in men vs women).
In this context, we consider strategies for learning robustly fair models under exclusive demographic,exclusive covariate, exclusive label and compound distribution shift scenarios represented in thecausal graphs of Fig. 6:
a) Exclusive demographic shift.
In Fig. 6(a), we consider an exclusive demographic shift throughthe direct path S
→ A .
In this case, the effects of S on Y and X are intercepted by A .
Therefore,training a fair model based on X (either via imposing equalized odds [64] or counterfactual invariancewith respect to A [72]) leads to a robustly fair model. b) Exclusive covariate shift.
In Fig.
6(b), we consider an exclusive covariate shift through the directpath S → X . In this case, there are multiple causal arrows into X that need to be addressed: A → Xfor fairness, and term S → X for robustness to distribution shift. In the case where some labeledtarget domain data is available, Schumann et al.
[61] implement separate independence regularizersto address each path, and demonstrate its effectiveness in settings that match this causal structure. Inthe fully unlabeled domain adaptation case, the shift is harder to account for. For example, the setof features that satisfy the selection criteria of Singh et al. [64] would only include A , leading to apredictor that relies solely on demographics data (Sec. B.3).
[Figure] [Figure] [Equation] A
Y
X
A
[Equation]
Y
[Equation]
S
A
Y X
A ⊥⊥ S
X ⊥⊥ S | Y
S
A
Y X
[Figure]
[Figure] [Figure]
[Equation]
Y
[Equation]
A X
Y
[Equation]
S
A X Y c) Exclusive label shift.
Lipton et al.
[37] propose a technique to detect an exclusive label shift, aswell as mitigate it through reweighting, requiring unlabelled target data.
This technique can be usedin addition to a fairness mitigation technique to lead to a robustly fair model.d) Compound shift.
Such a shift leads to multiple factors being affected by S , and current techniqueswould either be ineffective or degenerate to the trivial predictor.
While Fig.
6(d) represents the worstcase scenario, even simpler combinations of shifts would lead to similar results in practice.
Forinstance, adding a correlation between S and A to Fig.
6(b) leads to a compound shift.
In thisspeciﬁc case, mitigation might be possible by taking all intersections of A and S when using [61].
Inpractice, if the attribute or the shift have multiple discrete values, this will quickly become intractable,especially if distribution matching needs to be applied within each mini-batch.
When consideringfeature selection, [64] would return an empty set for both equalized odds or demographic parity,leading to a trivial predictor.
(Partial) mitigation might be obtained through the use of an adaptationtechnique as proposed in [66, 47].
B.2 Causal prediction tasks
We repeat the analysis above for the simplest causal prediction task in which X is a direct cause of
Y , and assuming that A is a direct cause of X and Y .
We then consider the following distributionshift scenarios represented in Fig.
a) Exclusive demographic shift.
As in the anti-causal case, the relationship between X and Y is not affected by S if we impose invariance of Y on A .
To this end, training a fair model based on theset { X, A } would lead to a robustly fair model [64].b)
Exclusive covariate shift.
In this setting, note that ( Y, A ) ⊥⊥ S | X .
Thus, fairness and robustnessto distribution shift guarantees can be obtained independently.
This setting corresponds to the classic covariate shift scenario treated in much of the domain adaptation literature, where potential solutions
[Table]
are to match the distributions of X across environments by strategies such as reweighting [e.g. 62] orinvariant representation learning [e.g. 6].
The advantage of such a techniques is that they only requireunlabelled target data.c)
Exclusive label shift.
While solutions are investigated for anti-causal prediction tasks, label shiftson causal prediction tasks are understudied and the absence of a direct path S
→ Y is often anassumption for mitigation techniques [64, 72].
To the best of our knowledge, there is no methodproposing formal fairness guarantees under exclusive label shift in causal prediction tasks.d) Compound shift.
As for the anticausal case, a compound shift would lead to insufﬁcient or trivialpredictors.
[Table]
We summarize the results for feature selection, and in particular the method proposed in [64] in
Table 1.
We observe that exclusive covariate and label shifts cannot be mitigated without excludingthe signal, leading to an empty set { V } in some of our simpliﬁed cases.
C Data and method availability
The dermatology data is not available to the public.
The de-identiﬁed EHR data is available based ona user agreement at Physionet [25].
The code for extracting Elixhauser and van Walren comorbidityscores from MIMIC-III is available at https://doi.org/10.5281/zenodo.821872 [30].
Wetake inspiration from the code made publicly by the authors of [70, 70] and available at https://github.com/google/ehr-predictions .
We use scikit-learn [51] to estimate the balancingweights.
D Computational resources
Our statistical tests include two main components:
• Estimation of P ( S | { V } ) : logistic regression or boosted gradient trees are trained times (number of bootstrap samples) for each test.
At a maximum, this operation uses 12Gb of RAM and 2 Gb of memory.
All such model trainings are performed in notebooks.• Summary of U : when the dimensionality l of U is larger than a couple of dozens, wesummarize U by training a model f ( U )
→ Y .
For dermatology, a model training and tuning requires 28 TPU resources over 24-30 hours.
For EHR, a model training requires 2 CPU for∼ 1 hour.
To assess model performance and fairness across multiple distribution shifts, we train 10 replicatesof a dermatology model f (
X , A )
→ Y on the source data, and 10 replicates of an EHR model f ( M, X, T ) → Y .
We also train 10 replicates for the joint training in dermatology.
The training of larger models is performed on an internal cluster.