Diagnosing failures of fairness transfer across distribution shift in real-world medical settings
Google Research
Subhrajit Roy
Google Research
Diana Mincu
Google Research Chen Google Google
Yuan Liu
Google Research
Vivek Natarajan
Google Research
Alan Karthikesalingam
Google Research
Google Research
Silvia Chiappa
DeepMind Google Research Abstract Diagnosing and mitigating changes in model fairness under distribution shift is an important component of the safe deployment of machine learning in healthcare settings.
Importantly, the success of any mitigation strategy strongly depends on the structure of the shift.
Despite this, there has been little discussion of how to empirically assess the structure of a distribution shift that one is encountering in practice.
In this work, we adopt a causal framing to motivate conditional independence tests as a key tool for characterizing distribution shifts.
Using our approach in two medical applications, we show that this knowledge can help diagnose failures of fairness transfer, including cases where real-world shifts are more complex than is often assumed in the literature.
Based on these results, we discuss potential remedies at each step of the machine learning pipeline
Introduction
As progress is made to integrate machine learning (ML) systems in real-world applications, one concern is how the ML model’s behaviour might be affected by changes in the data distribution.
While extensive work has investigated the effect of distribution shift on ML model performance [see 48, 77, 21, 74, for reviews], recent work has also highlighted that fairness properties that are satisﬁed on the data used to develop a model might not always hold if the data distribution changes [34, 31].
In high-risk domains like healthcare, this can translate to ‘if a model satisﬁes fairness criteria when trained in “Hospital A”, it may not satisfy them when tested in “Hospital B”’.
Fueled by this observation, a new ﬁeld of research is emerging to understand the relationship between ML fairness and robustness to distribution shift [e.g. 61, 1, 72, 16, 85, 65] and to provide mitigation techniques for obtaining robustly fair ML models [e.g. 64, 67, 66, 86, 15].
Importantly, the success of robustly fair machine learning depends strongly on the nature of the distribution shifts that the system will encounter in practice.
This point has been made elegantly in existing work as a motivation to use causal structure to provide fairness guarantees for speciﬁed distribution shifts [64, 67].
However, to the best of our knowledge, few methods can diagnose the nature of the shift and guide towards an appropriate mitigation strategy [66].
In this work, we design statistical tests that can, based on a simpliﬁed causal graph of an application and a small target dataset, assess the structure of the distribution shift encountered by the system and identify aspects of the data that hinder the transfer of fairness properties.
Using applications in dermatology and Electronic Health Records (EHR), we show that this analysis is crucial for selecting an appropriate mitigation strategy.
Further, our work also reveals that clinically plausible shifts are often more complex than can be handled by current mitigation techniques, highlighting the need for additional research into a broader set of remedies across the entire ML pipeline.
Background
Notation We consider problems where we are given variables X , A , Y , where X is a set of features,
A is one or more sensitive attributes, and Y is an outcome or label to be predicted.
Our goal is to build a classiﬁcation or regression model f
( X (cid:48) ) , where, depending on the context, X (cid:48) :
=
( X, A ) or X (cid:48) :
= X .
We are concerned with the fairness properties of f , both on the training distribution and in one or more deployed environments.
Depending on the setting, we focus on several statistical deﬁnitions of fairness [5]: gaps in subgroup accuracy, demographic parity, and equalized odds.
See
Sec.A.1 for their formulation.
Formally, we represent distribution shifts using the so-called Joint Causal Inference (JCI) framework of Mooij et al.
We represent causal relationships between elements of A , X , Y in a causal
Bayesian network (CBN, see Sec.
A.2 for an introduction).
In a CBN, a node U i with an arrow into U j is called a direct cause of U j .
We call all direct causes of a node U i its causal parents, and denote them pa
( U i ) .
To represent shifts, we augment the CBN with an environment variable
S , where the event S = 0 indicates data from the source environment and S = 1 denotes data from the target environment, so that source data are distributed according to P ( A, X, Y | S = 0) and the target data to P ( A, X, Y | S = 1) .
This is called the joint causal graph .
In the joint causal graph, arrows from S denote changes to statistical relationships that are induced by the shift.
Speciﬁcally, an arrow from S to some variable U indicates that the conditional distribution of that variable given its causal parents could change between training and deployment environments, while the absence of such an arrow would indicate that the conditional distribution remains the same, i.e.
P ( U | pa ( U ) , S = 0) =
P ( U | pa ( U ) , S = 1) .
Components of Distribution Shifts In JCI formalism, the components of a distribution shift are represented by direct arrows from the shift variable S to variables A , Y , and X .
Here, we brieﬂy summarize how these arrows can be interpreted.
S
→ A arrows represent demographic shifts , implying P ( A | S = s ) changes for different values of s .
An example of such a shift is a difference in the age distribution of patients in S = 0 versus S = 1 .
S
→ X arrows represent covariate shifts 3 , where the conditional distribution of input features P ( X | pa ( X ) , S = s ) changes for different values of s .
For example such a shift could result form using different cameras or views to acquire an image in S = 0 and S
= 1 (c.f., acquisition shift in Castro and Glocker [7]).
Finally, S → Y arrows represent label shifts 4 , where the conditional distribution of labels P ( Y | pa ( Y ) , S = s ) changes for different values of s .
A label shift can occur through different disease prevalences, or different strategies for obtaining labels, across S = 0 and S
= 1 (c.f., prevalence shift and annotation shift, respectively, in Castro and Glocker [7]).
In real-world applications, distribution shifts often include multiple components, where there are multiple arrows from S to { A, X, Y } .
We call such shifts compound .
Compound shifts often occur when there is an unobserved variable that affects multiple variables whose distribution changes between source and target.
An example is that of a new device on the market [23].
Such a device could create qualitatively different images X , but it could also change the observed patient population by being made more available to certain subsets of demographic groups that tend to be more or less healthy.
In this case, the shift induced by introducing the device simultaneously affects A, X and Y .
Shift Structure and Fairness Transfer
The transfer of fairness depends on the causal structure of the application and the structure of the shift [64, 65].
In particular, implications for fairness transfer can be directly read off of conditional independences with respect to the shift S in the joint causal graph.
For example, for classiﬁers that take a set of variables { V } as input, the following metrics are known to be invariant across the shift under the following conditional independence conditions:
• Classiﬁcation error ( P ( Y (cid:54) =
f ( V ) | V , S ) ) is invariant if Y ⊥⊥ S | { V } [39, 64].
• Demographic parity gap ( max a ∈A E [ f ( V ) | S, A
= a ] − min a ∈A E [ f ( V ) | S, A = a ] ) is invariant if { V } ⊥⊥ S
| A [64].
• Equality of odds gap ( max a,y ∈AY [ f ( V ) | S, A = a, Y
= y ] − min a,y ∈AY [ f ( V ) | S, A = a, Y
= y ] ) is invariant if { V } ⊥⊥ S | Y, A [64].
In JCI formalism, the above conditions imply that fairness transfer requires certain arrows from S to { A, X, Y } to be missing.
The nature of the shift also strongly inﬂuences which mitigation strategies are applicable.
We discuss implications for mitigation strategies in more detail in Sec.5.
Empirical tests for assessing shift structure
Given the central role of distribution shift structure in fairness transfer, our main methodological proposal is to investigate the structure of shifts empirically when fairness fails to transfer.
In this section, we outline a general strategy for testing which components of a shift are present, given data from source and target distributions, and a BCN for the application.
Approach.
For each variable U in the graph (i.e. each A , X and Y ), we assess whether there is a direct effect of the environment S on U .
We test the null hypothesis H 0 : P ( U | pa ( U ) , S = 0) =
P ( U | pa ( U ) , S = 1) .
Rejecting this hypothesis implies an arrow S → U in the joint causal graph.
If the marginal distribution of U differs across source and target distributions, this comparison can be challenging.
To isolate the direct effect of S , we compute weights that balance the causal parents pa ( U ) across environments [55, 28].
We then test the hypothesis that the reweighted distributions of U match across the environments S .
For simplicity, we test whether the reweighted means are equal, and in cases where the dimensionality of U is high, we conduct this test on functions of U that reduce the dimensionality, following strategies in Rabanser et al.
This procedure is described in Algorithm 1 and detailed in Sec. A.3.
Validation.
We conduct three validation experiments for our testing procedure using data from our dermatology application in Sec.
A.3.3.
We ﬁrst assess Type I error in an experiment where we test random splits of the same data against each other.
Our test displays a false positive rate similar to our threshold of 5% for hypothesis testing.
We however note that the variance of this result increases with smaller numbers of samples.
We then conﬁrm non-trivial power in an experiment where we introduce a shift by subsampling younger patients with a particular skin condition in the source dataset (a shift on A and Y ).
Our testing approach correctly identiﬁes the dimensions of A and Y that were affected by S due to the engineered shift.
We replicate this test for a shift in Y only, with the same result.
Distribution shifts in real-world healthcare applications
Here, we present two case studies from the healthcare domain in dermatology and in clinical risk prediction using Electronic Health Records in which fairness does not transfer (Fig. 1).
As per [32], we consider the setting of ‘dataset shift’, whereby a model is developed on the source data and tested on the target data 6 , which is a common setting in medical applications [78, 84, 32]. 5 A Algorithm 1 (Conditional) independence testing assessing the nature of shift S on a single variable U ∈ G .
U represents the feature values or its summary if high-dimensional.
(cid:12) is the Hadamard product.
Require: source dataset D , target dataset D (cid:48) , a joint causal graph G Split D into D w , D t ; Split D (cid:48) into D (cid:48) w , D (cid:48) t for 1: n bootstrap do Sample V from D n 0 × l t and V (cid:48) from D (cid:48) n 1 × l t Set w s = 1 n s for s ∈ { 0 , 1 } .
if pa ( U ) not ∅ then (cid:46) Compute balancing weights
Sample dataset Q = { pa ( U ) | S = 0 } from D w and Q (cid:48)
= { pa ( U ) | S = 1 } from D (cid:48)
w
Build predictor P ( S | pa ( U )) classifying Q from Q (cid:48)
w s ( pa ( U ))
∝ P ( pa ( U ) | S = s ) − 1 ∈ R n s for s ∈ { 0 , 1 } .
end if
Compute t-statistic between [ w 0 (cid:12) V ( U ); − w 1 (cid:12) V (cid:48) ( U )] andend for Compute p-value from t-test of bootstrapped t-statistics againstreturn p-value (cid:46) The p-value that U ⊥⊥ S | pa ( U )
[Figure]
4.1 Predicting skin conditions in dermatology
In this task, we predict common skin conditions (26 conditions plus an additional ‘other’ category to capture the long tail), based on one or multiple images of the pathology of interest and additional metadata [38].
The data used as source is a subset of the data described in Liu et al.
Brieﬂy, it consists of de-identiﬁed retrospective adult cases from a teledermatology service serving 17 sites from 2 states in the United States.
Each case contained 1-6 clinical photographs of the affected skin areas as well as patient demographic information and medical history (see Table S1 of [38]).
We split the source data such that it comprises 12,024 cases for training, 1,925 for validation and hyper-parameter tuning and 1,924 for testing.
We consider how the fairness properties of a model trained in this setting transfer to a target dataset, unobserved at model development time, consisting of 1,843 labelled teledermatology cases from
Colombia.
In the Supplement, we consider transfer to a second target from skin cancer clinics in
Australia, as well as the properties of a model jointly trained on the source and the second target, and deployed on the ﬁrst target data.
The model we consider is a ResNet architecture, as in [38, 57]. [Figure]
We assess model performance and fairness across 10 replicates of the model on the source and on the target (Fig. 1).
On the source data, the model performs similarly across all subgroups and could reasonably be assessed as ‘fair’.
However, the maximum gap between age groups increases starkly on the target (from 1 .
05% to 12 .
To understand the failure of fairness transfer in this context, we investigate the structure of the shift by referring to Algorithm 1.
We begin by drawing a simpliﬁed causal graph G of the dermatology task at hand (Fig.2(a)).
We consider this an anti-causal learning task [60], in which the skin condition label Y is a cause of the input image X , and consider demographic information A to be a causal parent of both X and Y .
We omit variables that are unobserved in either the source or target data.
We now test whether the shift S has direct effects on A , Y , or X .
Overall, we ﬁnd that the shift is compound, and affects all three aspects of the data distribution.
We begin with A , which comprises age and sex.
Because A has no causal parents in our graph (besides, potentially, S ) we directly assess differences in age and sex across environments using classical tests.
Age has a different distribution across the two datasets (see Fig. 2(b)), with the population in the source data being typically younger (median: 40 years old, 25 % quantile: 27, 75 % quantile: 54) than in the target data (median: 44 years old, 25 % quantile: 30, 75 % quantile: 59).
Sex distributions are well matched across the source and target.
These results suggest a direct effect of S on A , and we add this relationship to the causal graph (Fig. 2(a), in purple).
We then assess whether S directly affects the labels Y .
In our graph, A is a causal parent of Y , so following our testing strategy in Algorithm 1, we test whether P ( Y | A, S = 0)
=
P ( Y | A, S = 1) .
(Recall that conditioning on A isolates the direct effect of S on Y .)
Using 27 univariate tests (one per condition, [54]), we obtain signiﬁcant differences between P ( Y | A, S = 0) and P ( Y | A, S = 1) for 3 conditions ( p < 0 .
05 , Bonferroni corrected).
Fig.2(c) illustrates the label shift in a speciﬁc age and sex subgroup (here females aged 65+, 409 cases in the source, 128 in the target data).
We see that the source data includes more cases of ‘eczema’ and ‘psoriasis’ while conditions like ‘actinic keratosis’ and ‘seborrheic dermatitis’ are more represented in the target.
These results suggest that the environment also directly affects the labels (orange link in Fig.2(a)).
Finally, we consider whether the features of the images themselves (designated by X ) are directly affected by S .
Conditioning on causal parents of X , we test whether P ( X | Y, A, S = 0) =
P ( X |
Y, A, S = 1) .
We summarize X by training a model f ( X ) →
Y on the source, a strategy proposed in
Rabanser et al.
Our weighted tests suggest a signiﬁcant difference between the two distributions ( p < 0 .
05 corrected on 1 dimension).
Fig.2(d)
illustrates this difference by computing the number of images per case in the group of older females considered above with cases labelled as ‘SK/ISK’ ( source median: 3, n = 61 , target median: 5, n = 18 ).
This result suggests the existence of the direct path S
→ X
and we add this relationship to the graph (green link in Fig.2(a)).
Based on these results, we observe that all aspects of the data are affected by the environment.
Thus, none of the conditional independence conditions for fairness transfer hold in this setting.
However, we also see that some conditions might display invariance once controlled for demographics, which suggests possible mitigations that we explore in Section 5.
The Supplement includes experimental details and further results on the second target and the joint training.
4.2 Clinical risk prediction from Electronic Health Records
In this application, we predict the clinical outcomes of patients based on EHRs.
EHRs record the time series of interactions between a patient and the clinical system (e.g. medication, labs, vitals, ...).
Distribution shift is a major concern for systems that consume EHR data, which can be heterogeneous across centers [79, 65], and even within a single system [45].
As an example, over half of hospitals in the US with intensive care units (ICUs) only have a single unit where all critically ill patients (medical, surgical, cardiac, etc.) are admitted 9 .
Risk scores, such as the Acute Physiology and Chronic Health
Evaluation II (APACHE II) score are used to evaluate all critically ill patients, but do not perform equally across clinical subgroups, e.g. underperforming on patients with cardiac disease [53].
It would be desirable to avoid such issues with risk scores constructed through machine learning.
We use the open access, de-identiﬁed Medical Information Mart for Intensive Care III (MIMIC-III) dataset [29], which consists of data from admissions to ICUs at the Beth Israel Deaconess Medical
Center between 2001 and 2012.
This clinical system has the beneﬁt of having separate specialized
ICUs and allows to assess the generalizability of risk scores estimated in one clinical population to another.
Based on clinical input, we consider the Medical ICU (MICU), Surgical ICU (SICU) and
Trauma Surgical ICU (TSICU) to be generalist ICUs (source data); whereas the Cardiac Surgery
Recovery Unit (CSRU) and Coronary Care Unit (CCU) are specialized ICUs (target data).
This split leads to 17,641 patients included in the source dataset and 10,442 patients in the target dataset, after selecting adult patients with a length of stay of minimum one day and with a recorded care unit.
Our goal is to obtain a robustly fair model that predicts prolonged ICU stay (i.e. length of stay > 3 days, as in [75, 52]) using data from the ﬁrst 24 hours of ﬁrst ICU admission and a recurrent architecture [69, 70, 58].
The model is trained on 80% of the source data, tuned using a separate split of 10% and tested on the remaining 10% .
See the Supplement for further details.
Noteworthy, the model does not have access to the ‘reason for visit’.
Figure 1(b) shows that the fairness properties of this model, as measured by gaps in subgroup performance, demographic parity (DP) and equalized odds (EO), do not transfer across ICU types.
The model displays unfairness w.r.t.
age in both datasets (maximum gap ∼ 7% , similar DP and EO scores).
However, the pattern of unfairness is different in the two datasets, with e.g. the 45-64 years old subgroup being under-served in the source but over-served in the target.
The performance gap between sex groups increases from 1% in the source (DP: 0 .
002 ± 0 . 002 ) to 2 .
7% (DP: 0 .
016 ± 0 . 003 ) on the target.
To understand this failure of fairness transfer, we examine the structure of the shift.
First, we draw a causal graph for the variables in this problem, building on the work in Singh et al.
Here, we incorporate demographics A as age and self-reported sex, comorbidities M as previous medical history deﬁned by ICD-9 codes [9, 17, 27, 35], observed features X as all labs and vitals, and Y as our prolonged length of stay label.
As a more comprehensive feature set leads to better predictive [Figure] performance [58], we use all 59,351 features at our disposal and add ‘treatments’ T (e.g. medications) to the graph (Fig. 3(a)).
In our graph, S represents the ICU unit type that a patient is admitted to.
We now test whether the shift S has direct effects on the variables in this problem using our proposed approach.
Beginning with A , which has no causal parents, we test whether P ( A | S = 0) =
P ( A |
S = 1) .
We estimate the distribution of age and sex in each dataset (Fig. 3(b)).
We observe that the population in the source is typically younger than the population in the target (t-test: p < 0 . 0001 ).
In addition, the proportion of males in the source and target differ (source: 50%, target: 65%).
Therefore,
S → A .
To assess the shift’s effect on comorbidities M , we refer to Elixhauser and Vanwalraven scores [20, 71] computed on the source and target data [30] 10 .
Figure 3(c) displays the distribution of the 30 comorbidities for older males ( n = 3 , 951 in source and n = 3 , 588 in target).
As expected, even after adjusting for the causal parent A , patients in the target data have a higher prevalence of comorbidities related to the peripheral vascular system than in the source data and M (cid:54)⊥⊥ S | A forcomorbidities (weighted tests: p < 0 .
05 , Bonferroni corrected).
We repeat the analysis to assess the direct effect of S on the treatments ( P ( T | A, M, S = 0) =
P ( T | A, M, S = 1) : p < 0 . 05 ), on the labs and vitals ( P ( X | A, M, T, S = 0) =
P ( X | A, M, T, S = 1) : p > 0 . 05 ) and on the label ( P ( Y | A, M, T, S = 0) =
P ( Y | A, M, T, S = 1) : p > 0 . 05 ) in the Supplement.
Figure 3(d) displays the prevalence of prolonged length of stay for older males with a solid tumor comorbidity and receiving vasopressors/inotropes ( n = 55 in source and n = 16 in target).
Our results suggest that the shift is compound, i.e. the ICU unit a patient is admitted to changes the relationships between multiple variables in our causal graph.
We note that this conclusion is not surprising clinically, given that the environment S is correlated with the reason for their ICU admission, and that this reason for admission (unobserved) is a main driver for all variables.
As in the dermatology case, the compound shift leads to multiple paths for S and A to affect the covariates
M and X and the label Y , explaining the failure of fairness transfer.
Mitigation and related work
A variety of fairness, robustness and robustly fair mitigation strategies have been proposed.
We contextualize a number of methods in terms of anti-causal and causal learning tasks [60] in Sec.
B, and highlight how understanding the nature of the distribution shift can guide mitigation.
Fairness mitigation.
Pre-, in- and post-processing techniques for mitigating unfairness have been proposed [see 5, 46, for overviews].
Fairness metrics and mitigation techniques currently used in real-world applications are typically evaluated on a single learning task or data distribution [e.g.
More similar to our settings, the work by Creager et al.
[15] addresses fairness in dynamical systems, although only for low-dimensionality variables.
Based on our causal framing, applying fairness mitigations in the source domain could potentially aid in fairness transfer by cutting some edges in the underlying CBN [72].
However, fairness mitigation on the source only leads to provable fairness transfer in speciﬁc settings, i.e. mostly when the shift is demographic (Sec.B).
Based on this analysis, performing fairness mitigation in the presence of a compound shift would fail to provide a fair result on the target.
Considering the applications above, a reasonable option would be to perform fairness mitigation for the EHR task based on age.
We perform post-processing of the predictions to enforce either demographic parity or equalized odds across age subgroup using the method in Alabdulmohsin and Lu ˇ ci ´ c
This results in a decrease in demographic parity to value 10 − 5 in the source, but to (slightly) increase to value 0 .
078 ± 0 .
025 in the target data.
Similarly, equalized odds decrease to value 10 − 5 and the gap in model accuracy between groups is reduced from 7% to 0.8% (Fig. 4) in the source, but increases from 0 .
05 ± 0 .
008 to 0 .
16 ± 0 .
04 on the target data, and the gap between age groups increases to 11.4%.
We therefore observe that mitigating for either demographic parity or equalized odds on the source leads to no improvement, or even worsening of the fairness properties on the target, as predicted by our analysis.
Robustness mitigation: Relations between robustness guarantees and individual and group fairness have been studied in [82, 44, 80, 81] and [72] respectively.
[1, 16, 85] have investigated the relationship between fairness and distribution shift experimentally by studying how robustness mitigation strategies affect fairness metrics.
These works do not provide provable robustly fair models.
Robustly fair mitigation.
Work explicitly investigating fairness properties under distribution shift can mostly be divided into methods that provide a priori guarantees or bounds [e.g. 61, 64, 67] and methods that adapt to the new distribution [e.g. 13, 47, 66, 86, 31, 18, 68].
Recent work by Singh et al.
[64] suggests a straightforward mitigation with guarantees, based on selecting features that satisfy the fairness transfer criteria outlined in Section 2.
However, this approach is limited under compound shifts with complex data.
Most notably, it cannot be used in cases where there is a label shift ( S → Y ).
However, even when there is no label shift, under compound shifts the criteria for feature selection can yield a trivial predictor where no observed variables can be used.
This is the case in our EHR example under the joint causal graph in Fig. 3(a).
We discuss the applicability of [64] across different causal scenarios in Sec.
B.3.
In addition, feature selection approaches are more suited for semantic features and, while it is technically feasible, would not lead to insightful results in non-semantic features such as pixels.
In computer vision, masking has been used previously as an approximation to feature selection to constrain the learning towards speciﬁc areas of the image [56, 63].
While this technique has shown some promise with chest x-rays [73], it is impractical due to the additional requirement of a mask for all training images.
In addition, it increases the ratio of signal of interest compared to undesirable signal, but does not exclude such signals and comes with no guarantee.
Another line of work considers adapting transfer learning and meta-learning algorithms to consider fairness constraints and therefore ensure generalization to new tasks also in terms of fairness [13, 47, 66, 86].
Speciﬁcally, Coston et al.
[13] modify the standard approach to transfer learning under covariate shift by adding a constraint to the weight learning objective that enforces closeness between all pairs of protected groups.
Schumann et al.
[61] provide bounds on fairness transfer by adding adversarial heads on both the in-source attribute A and the environment S .
When we cast this work into our causal framework, we notice that it would provide a robustly fair model under an exclusive covariate shift in an anti-causal learning task.
Slack et al.
[66] propose meta-learning with a regularizer that imposes demographic parity or equalized odds to improve the transfer of fairness.
The method requires a small set of labelled target data (K-shot learning) and multiple tasks, and comes with no guarantees.
Work on ﬁne-tuning and transfer learning [47, 86] leverages task similarity to learn fair representations that provably generalize to unseen tasks, similarly requiring a small set of labelled target data.
Importantly, these methods generate one tuned model per environment, which is not suitable for all real-world deployments of ML.
It is also unclear whether these techniques could handle more complex tasks, e.g. predictions from sparse multivariate timeseries as in the case of
EHR.
Shift detection.
Many methods exist to detect covariate shift [e.g. 40, 8] or label shift [37, 24].
They however often rely on the assumption that other variables are not affected by the shift.
For instance,
Lipton et al.
[37] relies on the assumption that X ⊥⊥ S | Y .
On the other hand, Slack et al.
[66] introduce a technique to identify mean shifts that could lead to unfairness.
While the method is an interesting step towards ‘fairness transfer warnings’, this strategy does not allow to guide the selection of mitigation.
Recently, Singh et al.
[65] investigate differences in source and target distributions using squared maximum mean discrepancy [26], but do not take the causal structure into account.
To differentiate direct from indirect effects of S on variables U , they perform causal inference and visually inspect the obtained graph.
This method can only be applied to low-dimensionality datasets.
Discussion
In this work, we applied statistical tests within a joint causal framework [43] to better understand failures of fairness transfer across distribution shift.
Our approach tests for direct effects of the environment on every random variable in a task.
We use this technique to characterize distribution shifts that result in two real world failures of fairness transfer in the healthcare domain.
We show that this knowledge can then guide the choice of an appropriate mitigation strategy, when applicable.
Our work highlights real-world challenges that prevent the development of robustly fair models.
Given that each task leads to different complexities in terms of causal graph, expected shifts, and fairness requirements, these ﬁndings support looking beyond purely algorithmic solutions.
Considering the whole ML pipeline, we can re-interpret the remedies discussed in Chen et al.
[10] for fairness transfer:
Problem selection.
Consider focusing on tasks with lower anticipated harms or clinical/policy safeguards against ML unfairness.
Data collection.
Consider early data collection from anticipated deployment environments in which distribution shifts are suspected.
This would allow to perform (conditional) independence tests to guide further mitigation.
Outcome deﬁnition.
Consider intermediate outcomes for which unfairness under distribution shift might be less consequential (e.g. image segmentation compared to diagnosis).
Algorithm development.
Based on (2), some strategies can be used if their assumptions are satisﬁed.
In addition, in the present work we used a causal framework.
However, techniques that are not easily cast into this framework exist.
For instance, manual feature engineering has been used to improve the robustness of EHR models to distribution shift [45].
While this
work has not evaluated the fairness of the model in the source and target distributions
, we believe that feature engineering might be a promising avenue to enforce speciﬁc inductive
biases during learning while reducing the risks of distribution shifts.
We however note that our work does not
assess the impact of model architecture or different training strategies which
might lead to compounded
Post-deployment.
Further safeguards may be envisaged to safely deploy ML systems, e
g. with a prospective observational (non-interventional) integration [23].
Transfer learning or retraining with site-speciﬁc updates might then be considered as necessary prior to interventional deployment.
Finally, those deployments may be accompanied with robust post-market surveillance to continuously monitor fairness
Limitations and future work.
The power of our diagnosis method depends on the number and quality of the samples.
Therefore, it should be perceived as displaying potentially problematic aspects of the data, rather than assessing invariant features.
This is a common limitation in shift detection [37] and mitigation [64].
Furthermore, the data at hand usually represents a set of proxies for the underlying variables.
Therefore, even if a shift seems to satisfy some invariance assumptions, there are no sharp guarantees [22].
This is especially true for demographic factors: features such as ethnicity, age or sex only approximate sensitive attributes.
We are also limited by which factors are observed across all environments and cannot assess changes in unobserved variables (e.g. social determinants of health).
While our work covers different types of shift, future work should investigate further sources of shift and biases, such as sampling biases or missingness [7, 85].
In addition, extending the testing strategy to compare multiple (discrete or continuous) versions of the environment S would be valuable.
By specifying causal graphs for each problem, we make assumptions about the relationships between variables.
These assumptions might not represent the true underlying data generation process.
In addition, the effects of A could potentially be divided into ‘fair’ and ‘unfair’ effects, as proposed in [12, 42].
For real-world applications, further assumptions were made, e.g. considering the silver standard label in dermatology as a good approximation for the gold standard label (hence an anticausal task)
The value of these graphs is purely illustrative and should not be considered for medical applications without validation.
In terms of metrics, we have focused mostly on demographic parity and equalized odds, due to their recent causal grounding [72].
Different metrics might, however, be affected differently, and it is possible that some metrics are more or less ‘sensitive’ to distribution shift.
Similarly, different fairness mitigation strategies might be affected differently by distribution shift, although none provides guarantees under distribution shift.
We also note that ‘fairness’ in the healthcare domain is an active discussion, and that equalized odds or demographic parity metrics do not consider factors such as social determinants of health or health equity.
As discussed in [52], considering human elicited metrics might be an avenue forward.
Similarly, ‘fairness’ might change depending on the context [59] and different metrics might be relevant in different environments
Acknowledgments and Disclosure of Funding
The authors would like to acknowledge and thank Lucas Dixon, Noah Broestl, Sara Mahdavi, Nenad
Tomasev, Cameron Chen, Stephen Pfohl, Matt Kusner, Victor Veitch, Jon Deaton, Shannon Sequeira,
Abhijit Guha Roy, Jan Freyberg, Aaron Loh, Martin Seneviratne, Patricia MacWilliams, Yun Liu, Christopher Semturs, Dale Webster, Greg Corrado and Marian Croak for their contributions to this effort. This work was funded by Google. Appendix A Methods
In this section, we provide further details on the methods, including the mathematical deﬁnitions of our metrics and an in-depth description of our testing procedure.
A.1 Fairness metrics
Our work focuses on statistical group deﬁnitions of fairness [5].
In particular, we frame the discussion around demographic parity in which the model’s output is statistically independent of the sensitive attribute, i.e. f ( X (cid:48) ) ⊥⊥ A , and equalized odds in which the independence holds conditionally on the outcome, i.e. f ( X (cid:48) )
⊥⊥ A | Y .
In the experiments, these metrics are computed for binary tasks.
We also compute the gap in subgroup accuracy.
For all fairness metrics, the lower, the better.
We deﬁne demographic parity for a predictor f as [19, 83, 41, 3]:
[Equation]
Equalized odds is computed in a similar fashion by conditioning on the positive and negative classes, taking the average of the discrepancies across classes.
Subgroup performance is computed as the accuracy (top-1 or top-3 in the case of dermatology, accuracy for EHR) within each subgroup.
When the sensitive attribute is non-binary, we compute the maximum difference between any pair of subgroups.
A.2 Causal framework
A Bayesian network [14, 33, 49, 50] is a directed acyclic graph (DAG)
G whose nodes X 1 , . . .
, X D represent random variables and links express statistical dependencies among them.
Each node X d is associated with a conditional probability distribution (CPD) p ( X d | pa ( X d )) , where pa ( X d ) denote the parents of X d , namely the nodes with a link into X d .
The joint distribution of all nodes is given by the product of all CPDs, i.e. p ( X 1 , . . .
, X D | G ) =
(cid:81)
Dd =1 p ( X d | pa ( X d )) .
This function is assumed to be invariant across distributions.
A path from X i to X j is a sequence of linked nodes starting at X i and ending at X j .
A path is called directed if the links point from preceding towards following nodes in the sequence.
A node X i is an ancestor of a node X j if there exists a directed path from X i to X j .
In this case, X j is a descendant of X i .
We say that a set of nodes { X i , . . .
, X j } d-separates two random variables U and W if U ⊥⊥ W | { X i , . .
X j } .
A causal Bayesian network is a Bayesian network in which a link expresses causal inﬂuence rather than statistical dependence.
In causal Bayesian networks, directed paths are called causal paths .
A.3 Assessing the causal structure of shifts A.3.1 General Strategy
To assess the causal structure of a shift, we examine whether there is a direct effect of the shift S on a focal variable U .
This requires controlling for all other pathways by which U may depend on S .
To do this, we select a set of variables V such that { V } blocks all indirect paths from S to U .
We then test the following equality in conditional distributions: P ( U | V , S = 0) =
P ( U | V , S = 1) almost everywhere.
To test the equality of conditional distributions, we reduce the problem to testing whether the means of the two marginal distributions of U are equal when the distributions of the background variables
V are adjusted to follow the same distribution.
Using observed data from environments S =and S = 1 , we can perform such an apples-to-apples comparison using importance weighting.
In particular, for any distribution π ( V ) ,
[Equation]
for almost all V
[Equation]
Deﬁne the weights w 0 ( V ) :
= π ( V ) P ( V | S =0) , and w 1 ( V ) :
= π ( V ) P ( V | S =1) .
This result shows that we can test the equality of the conditional distributions by testing the implication that the mean of weighted outcomes is the same.
This leaves a degree of freedom for choosing the test distribution π ( V ) on which the conditional distributions will be compared.
This distribution then determines the weighting scheme that will be used.
We deﬁne π ( V ) to be a constant, i.e., uniform.
Then the weights are
[Equation]
and w
[Equation]
= 1) − 1 .
This weighting scheme is known as inverse probability weighting or IPW [55, 28], and is a popular choice.
We however note that other weighting schemes could be considered, e.g. overlap weights [36] or permutation weighting [4].
In practice, the likelihoods P ( V | S = s ) used to deﬁne the weights need to be learned from data.
Usefully, because of how the weights are normalized, the likelihoods can be replaced with classiﬁcation scores that estimate P ( S = s | V ) , leading to estimated weights ˆ w 0 and ˆ w 1 .
The choice of the classiﬁer can be informed by the expected relationships between { V } and S : for instance, logistic regression can be considered in the case of simple relationships, while more expressive classiﬁers such as gradient boosted trees can be considered in the case of non-linear relationships [4].
As discussed in Arbour et al.
[4], all classiﬁers are tuned (C for logistic regression, number of trees, tree depth and learning rate for boosted trees) in a cross-validation setup.
We then use a standard t-test to test the null hypothesis
[Equation]
To decrease the variance in the obtained t-statistics, we bootstrap the weighting and testing procedure.
The ﬁnal p-value associated with H 0 : P ( U | V , S = 0) =
P ( U | V , S = 1) is the p-value of a two-sided Z -test on the t-statistics obtained from each bootstrap.
Observations and tips From our observations, we note that it is best to allocate the largest available sample to estimate the likelihood ratio, compared to performing the ﬁnal statistical test.
We also observed that gradient boosted trees can lead to extreme weights in some cases.
This results in an increased variance and non-signiﬁcant results.
We therefore clipped the weights to a maximum of(arbitrary choice) to mitigate this effect.
A.3.2 Testing with High-Dimensional U
If U has low dimensionality, Rabanser et al.
[54] show that multiple one-dimensional tests can be used with correction for multiple comparisons (here Bonferroni).
On the other hand, if U has higher dimensionality (e.g. an image), summaries of U can be constructed and the test can be performed on these lower-dimensional summaries.
The validity of this approach follows from the fact that if the conditional distributions of U are the same, then so will the conditional distributions of any summary f ( U ) .
The trade-off is that the test loses all power to detect distributional differences that are compressed out by the summary f ( U ) , or to highlight speciﬁc dimensions in U that are more or less affected by S .
To deﬁne a summary that is relevant to the problem, Lipton et al.
[37] suggest deﬁning f ( U ) to be the output of a model the predicts some variable of interest (say the outcome Y ) using U .
As reported in Rabanser et al.
[54], other summarizing techniques could be considered.
A.3.3 Sanity checks with engineered shifts
We assess the speciﬁcity and sensitivity of our testing procedure (see Algorithm 1) using engineered shifts on the dermatology data.
To estimate Type I error, we compare the source data to itself, which should lead to all variables being independent of the environment.
More speciﬁcally, we select 10 ,random samples from the dataset to compute the weights, and another 1 , 000 random samples to apply the weights on and perform the weighted test.
These sets are boostrapped 100 times (separately for the source and target sets) and we assess the proportion of false positives that is obtained for each variable U .
We expect that we will obtain 5% of false positives, as deﬁned by our hypothesis testing threshold.
As we do not expect any relationship between { V } and S , we use a logistic regression to estimate P ( S = s | V ) .
When comparing P ( Y | A, S = 0) to P ( Y | A, S = 1) , we observe that the accuracy of the classiﬁer mirrors the chance level of 50% and that the proportion of false positives varies between 2 and 8 .
5% (Figure 5(a)).
Similarly, comparing P ( X | Y, A, S = 0) to P ( X | Y, A, S = 1) when using a model f ( X ) → Y to summarize X leads to a proportion of false positives between 2 and 7% (Figure 5(b)).
As the number of samples used to compute the weights is an important component of our testing approach, we repeat the process while varying the number of samples from 100 to 5,000.
Figure 5(c) displays that the variance on Type I error (here across conditions) decreases when the number of samples to ﬁt the P ( S = s | V ) classiﬁer increases.
[Figure]
[Table] [Table] To assess sensitivity, we then engineer a shift on the data that discards samples based on their age and condition (here Acne).
More speciﬁcally, we subsample data in order to decrease the bias for younger age in the Acne condition, leading to an increase in age for Acne samples of approximately 10 years.
Given that Acne is one of the most prevalent conditions in the dataset, we expect a signiﬁcant change on age, as well as on Y .
On the other hand, the images are not modiﬁed and hence
P ( X | Y, A, S = 0) and P ( X | Y, A, S = 1) should not display signiﬁcant changes.
We observe a signiﬁcant change in age between S = 0 and S = 1 .
There is no signiﬁcant effect of S on other demographic factors.
We do observe a direct effect of S on Y , with a strong effect on Acne as expected.
On the other hand, no tests display signiﬁcant differences when comparing
P ( X | Y, A, S = 0) with P ( X | Y, A, S = 1) .
This latter result was obtained with both a logistic regression and gradient boosted trees (given the more complex relationship between age and Acne).
Therefore, the testing approaches produces a causal graph that mirrors the graph used to engineer the shift, and our procedure is considered faithful.
Finally, we randomly shufﬂed the labels y k across the 27 conditions, for each case independently.
At the population level, this results in a change in P ( Y ) .
As expected, there was no effect of S on
A .
For Y , we observed signiﬁcant differences in P ( Y | A, S = 0) and P ( Y | A, S = 1) forconditions out of 27 (Bonferroni corrected).
Thanks to the correction, and given that the images were not modiﬁed, there are no signiﬁcant differences between P ( X | Y, A, S = 0) and P ( X | Y, A, S = 1) .
Overall, our tests are sensitive enough to detect the presence of changes on the affected dimensions, while not displaying an excessive amount of false positives.
B Causal framing of related works
In this section, we review how various transferable fairness approaches that have been proposed in the literature interact with different causal structures of domain shift depicted in Figures 6 and 7.
We assume the same sensitive attributes A in the source and target environments, and consider different relationships between the environment and other variables.
As in [7, 72], we split the analysis into causal or anti-causal prediction tasks [60].
A prediction task is causal if the features X are causes of the outcome Y , and anti-causal if the outcome Y is a cause of the features X .
Many risk prediction problems are causal, where risk factors may cause an adverse outcome, while many diagnostic problems are anti-causal, where the underlying disease may cause symptoms [7].
A takeaway from this review is that most methods are tailored to particular restricted shift structures, and their useful properties can break down under compound shifts.
B.1 Anti-causal prediction tasks
First, we consider anti-causal prediction tasks, in which the outcome Y is a direct cause of the features X , and that the sensitive attribute A is a direct cause of X and Y (the reality will likely be more complex and involve other variables in the paths (observed or not)).
The assumption that A is a cause of X and Y is a reasonable approximation in healthcare settings: e.g. on average, skin images from men are more hairy than those of women, breasts are visible in chest x-rays, and comorbidities are different across sexes and age ranges.
Sensitive attributes can also be related to the label, as prevalence can vary across subgroups (e.g. baldness is more prevalent in older patients) and some conditions might present differently across attributes (e.g. a heart attack in men vs women).
In this context, we consider strategies for learning robustly fair models under exclusive demographic, exclusive covariate, exclusive label and compound distribution shift scenarios represented in the causal graphs of Fig. 6:
a) Exclusive demographic shift.
In Fig. 6(a), we consider an exclusive demographic shift through the direct path S
→ A .
In this case, the effects of S on Y and X are intercepted by A .
Therefore, training a fair model based on X (either via imposing equalized odds [64] or counterfactual invariance with respect to A [72]) leads to a robustly fair model.
b) Exclusive covariate shift.
In Fig.
6(b), we consider an exclusive covariate shift through the direct path S → X .
In this case, there are multiple causal arrows into X that need to be addressed:
A → X for fairness, and term S → X for robustness to distribution shift.
In the case where some labeled target domain data is available, Schumann et al
[61] implement separate independence regularizers to address each path, and demonstrate its effectiveness in settings that match this causal structure.
In the fully unlabeled domain adaptation case, the shift is harder to account for.
For example, the set of features that satisfy the selection criteria of Singh et al.
[64] would only include A , leading to a predictor that relies solely on demographics data (Sec. B.3).
[Figure] [Figure]
[Equation]
S
A
Y
X
A ⊥⊥ S
Y ⊥⊥ S
S
A
Y
X
A ⊥⊥ S
X ⊥⊥ S | Y
S
A
Y
X
[Figure]
[Figure] [Figure]
[Equation]
A
[Equation]
Y
A
[Equation]
Y
[Equation]
S
A
X
Y
A
[Equation]
S
X
[Equation]
S
S A X Y c) Exclusive label shift.
Lipton et al.
[37] propose a technique to detect an exclusive label shift, as well as mitigate it through reweighting, requiring unlabelled target data.
This technique can be used in addition to a fairness mitigation technique to lead to a robustly fair model. d) Compound shift.
Such a shift leads to multiple factors being affected by S , and current techniques would either be ineffective or degenerate to the trivial predictor.
While Fig.
6(d) represents the worst case scenario, even simpler combinations of shifts would lead to similar results in practice.
For instance, adding a correlation between S and A to Fig.
6(b) leads to a compound shift.
In this speciﬁc case, mitigation might be possible by taking all intersections of A and S when using [61].
In practice, if the attribute or the shift have multiple discrete values, this will quickly become intractable, especially if distribution matching needs to be applied within each mini-batch.
When considering feature selection, [64] would return an empty set for both equalized odds or demographic parity, leading to a trivial predictor.
(Partial) mitigation might be obtained through the use of an adaptation technique as proposed in [66, 47].
B.2 Causal prediction tasks
We repeat the analysis above for the simplest causal prediction task in which X is a direct cause of
Y , and assuming that A is a direct cause of X and Y .
We then consider the following distribution shift scenarios represented in Fig.
a) Exclusive demographic shift.
As in the anti-causal case, the relationship between X and Y is not affected by S if we impose invariance of Y on A .
To this end, training a fair model based on the set { X, A } would lead to a robustly fair model [64].
b) Exclusive covariate shift.
In this setting, note that ( Y, A ) ⊥⊥ S | X .
Thus, fairness and robustness to distribution shift guarantees can be obtained independently.
This setting corresponds to the classic covariate shift scenario treated in much of the domain adaptation literature, where potential solutions
[Table]
are to match the distributions of X across environments by strategies such as reweighting [e.g. 62] or invariant representation learning [e.g. 6].
The advantage of such a techniques is that they only require unlabelled target data.
c) Exclusive label shift.
While solutions are investigated for anti-causal prediction tasks, label shifts on causal prediction tasks are understudied and the absence of a direct path S → Y is often an assumption for mitigation techniques [64, 72].
To the best of our knowledge, there is no method proposing formal fairness guarantees under exclusive label shift in causal prediction tasks. d) Compound shift.
As for the anticausal case, a compound shift would lead to insufﬁcient or trivial predictors.
B.3 Feature selection mitigation strategies
We summarize the results for feature selection, and in particular the method proposed in [64] in
Table 1.
We observe that exclusive covariate and label shifts cannot be mitigated without excluding the signal, leading to an empty set { V } in some of our simpliﬁed cases.
C Data and method availability
The dermatology data is not available to the public.
The de-identiﬁed EHR data is available based on a user agreement at Physionet [25].
The code for extracting Elixhauser and van Walren comorbidity scores from MIMIC-III is available at https://doi.org/10.5281/zenodo.821872 [30].
We take inspiration from the code made publicly by the authors of [70, 70] and available at https: //github.com/google/ehr-predictions .
We use scikit-learn [51] to estimate the balancing weights.
D Computational resources
Our statistical tests include two main components:
• Estimation of P ( S | { V } ) : logistic regression or boosted gradient trees are trainedtimes (number of bootstrap samples) for each test.
At a maximum, this operation usesGb of RAM and 2 Gb of memory.
All such model trainings are performed in notebooks. • Summary of U : when the dimensionality l of U is larger than a couple of dozens, we summarize U by training a model f ( U ) →
Y .
For dermatology, a model training and tuning requires 28 TPU resources over 24-30 hours.
For EHR, a model training requires 2 CPU for ∼ 1 hour.
To assess model performance and fairness across multiple distribution shifts, we train 10 replicates of a dermatology model f
( X, A )
→ Y on the source data, and 10 replicates of an EHR model f ( M, X, T ) → Y .
We also train 10 replicates for the joint training in dermatology.
The training of larger models is performed on an internal cluster.