Fairness-aware Missing Data Imputation
Abstract
Missing values are ubiquitous in real-world datasets and are known to cause unfairness in a machine learning algorithm’s decision-making process.
However, there has been limited work that aims to mitigate the unfairness associated with missing data imputation.
In this paper, we first derive a positive information-theoretic lower bound for the imputation fairness when using ground-truth conditional distribution for missing data imputation.
Furthermore, we propose a novel missing data imputation model, known as fairness-aware imputation GAN (FIGAN), which provides accurate imputations while achieving imputation fairness.
Through experiments, we illustrate that FIGAN can significantly improve imputation fairness, compared to the existing imputation methods.
At the same time, FIGAN can also achieve competitive imputation accuracy.
Introduction
Missing values frequently occur in real-world datasets.
While it has been well known that the absence of values will harm the performance of downstream predictions, recent works have connected the missing data with discrimination of decisions made by the downstream machine learning algorithms [Martínez-Plumed et al., 2019, Wang and Singh, 2021, Jeong et al., 2022, Zhang and Long, 2021b].
Investigations on the discrimination of algorithm’s decisions, namely algorithmic fairness, have been at the center of the trustworthiness of artificial intelligence.
Despite that there have been many works that aim to improve the fairness of a prediction model, there have been limited solutions that address the fairness concerns raised by the missing data.
In the presence of missing data, two classical solutions are: (1) drop the samples that contain missing values.
(2) impute the missing values.
To the best of our knowledge, there has not been any work that aims to improve fairness when one adopts an imputation model to handle the missing values.
In addition, there has not been any work that provides theoretical analysis on the fairness of imputation models.
In this paper, we first provide a theoretical analysis of imputation models that are correctly specified, in terms of the imputation fairness risk [Zhang and Long, 2021b].
In a more general setting, we propose fairness-aware imputation GAN (FIGAN), a GAN-based imputation model that can effectively control imputation unfairness.
Through experiments, we illustrate the outstanding performance of FIGAN on both synthetic and real datasets.
1.1 Related works
Algorithmic fairness focuses on the existence of discrimination during the decision-making process of a machine learning algorithm.
The existing work can be divided into group fairness [Feldman et al., 2015, Kamiran and Calders, 2009, Kamishima et al., 2011] and individual fairness [Dwork et al., 2012].
Group fairness emphasizes that members from different sensitive groups should be treated similarly by the algorithm, while individual fairness emphasizes that similar individuals should be treated similarly.
In this work, we focus on the group fairness of the missing data imputation methods.
We adopt the imputation fairness defined in Zhang and Long [2021b] as our fairness notion.
Recently, a line of research studied the relationship between missing data and algorithmic fairness.
Martínez-Plumed et al.
[2019] analyzed different causes of missing values, and empirically discovered that missing values may harm the fairness of the downstream prediction tasks.
Following this idea,
Wang and Singh [2021] conducted more extensive empirical evaluations and demonstrated that applying reweighting techniques can help mitigate the unfairness of the downstream tasks.
Goel et al.
[2021] established a causal framework to analyze the effect of missing data on the fairness of downstream tasks.
Later on, Jeong et al.
[2022] proposes a tree-based method for fair prediction in the presence of missing data, which does not require any missing data imputation.
Zhang and
Long [2021a] provided the first known theoretical results on fairness guarantee in the analysis of incomplete data.
However, as arguably the most popular approach for handling missing data, missing data imputation has not been well-studied in terms of algorithmic fairness.
To the best of our knowledge, the only work that studies the fairness of imputation methods is Zhang and Long [2021b], in which the authors assessed the fairness properties of popular imputation models through extensive numerical experiments.
In this paper, we are among the first to consider the problem of improving the fairness of missing data imputation and propose a novel imputation method that can yield accurate imputation while guaranteeing imputation fairness.
There is a large body of literature on missing data imputation.
Classical imputation approaches include using mean values of observed data, chained equations [Van Buuren and Oudshoorn, 1999], matrix completion [Mazumder et al., 2010] and EM algorithm
[Dempster et al., 1977].
With the recent development of machine learning, tree-based methods [Stekhoven and Bühlmann, 2012], optimal transport [Muzellec et al., 2020] and deep-learning-based methods [Yoon et al., 2018, Dai et al., 2021, Li et al., 2019, Bansal et al., 2021, Tashiro et al., 2021, You et al., 2020] are adopted for imputations.
Our proposed imputation model is based on the Generative Adversarial Networks (GAN)
[Goodfellow et al., 2020].
While there has been recent work on developing GAN models for fair data generation [Xu et al., 2018, 2019b,a], to the best of our knowledge, our work proposes the first known fairness-aware imputation model that leverages the power of GAN.
Preliminaries 2.1 Problem setup
We consider the problem of imputing missing values in dataset X
= { X i } ni =1 ∈ R n × p .
Each sample X i =
{ X i 1 , . . .
, X ip } is associated with a sensitive attribute A
= { A i } ni =1 (Figure 1).
Without loss of generality, we assume that there are two sensitive groups, that is, A i ∈ { 0 , 1 } .
We divide the features into two groups: X =
X miss ∪ X obs (Figure 1), where X miss =
( X missij ) denotes the subset of features that have missing values, and X obs =
( X obsij ) denotes the subset of features that are fully observed.
We define a missing data indicator M =
( m ij ) ∈ { 0 , 1 } n × p where m ij = 1 { X ij is missing } .
In addition, we define M i = 1 { X i has missing value } (Figure 1).
The missing data patterns can be classified into three mechanisms: missing completely at random (MCAR), missing at random (MAR) and missing not at random (MNAR)
[Little and Rubin, 2019].
Data are said to be MCAR if the missingness is independent of both observed and missing values, i.e.,
M ⊥ X ; Data are said to be MAR if the missingness only depends on observed values; Data that are not MCAR or MAR are said to be MNAR, i.e., M ̸⊥ X miss .
We further denote the imputation model of interest by ˆ X = f imp ( X obs , A ) .
A popular and useful metric that evaluates the performance of an imputation model is the mean squared imputation error (MSIE):
[Equation]
To assess the fairness of an imputation model, we define the MSIE within each sensitive group:
MSIE [Equation] ,j
[Equation]
aij where X a = ( X aij ) denote the samples within sensitive group
A = a .
Equipped with these definitions, we formally introduce the fairness notion for the imputation process. X X X
X
X
X M X obs complete cases completedata missing data observed data ( m ij = 1) ( m ij = 0) A
X
X
Xcomplete cases in sensitive group A =
[Figure]
in sensitive group A =
[Figure]
[Figure]
i
[Figure]
[Figure]
Definition 2.1 (Imputation Fairness) .
For a given imputation model f imp , the imputation fairness risk (IFR) is defined as
[Equation]
Intuitively, the imputation fairness risk (IFR) measures the difference in imputation error between groups defined by a sensitive attribute.
A perfectly fair imputation should provide equally accurate imputations for samples from different groups defined by a sensitive attribute.
2.2 A lower bound for IFR: when true imputation model is used
Conceptually, an imputation model learns the distribution of missing data X miss , given the observed information X obs and A .
That is, the imputation model f imp ( X obs , A ) approximates the conditional distribution P ( X miss | X obs , A ) .
While the true conditional distribution P ( X miss | X obs , A ) seems to be an ideal imputation, we argue that there can be inherent imputation fairness risk associated.
Indeed, in the following theorem, we provide an information-theoretic lower bound of the IFR for single-pattern missing data (i.e., for each sample, either it is fully observed, or the entire row in X miss will be missing simultaneously), when using the true conditional distribution P ( X miss | X obs , A ) for imputation.
Theorem 1.
For a dataset that has single-pattern missing data (i.e., m miss ij ≡ M i ), assume the true conditional distribution P ( X miss | X obs , A ) is adopted for imputation, then imputation fairness risk has the lower bound:
[Equation]
where K is the number of features that has missing values (i.e., the number of columns in X miss ), and
[Equation]
is the conditional variance of the missing data in the j -th column of X miss .
The proof of the theorem is provided in Appendix A.1.
Our results imply that even if the ground truth conditional distribution is used for imputation, there exists a positive information-theoretic lower bound for the imputation fairness risk.
In practice, the true conditional distribution is not accessible.
However, it is possible to correctly specify the conditional distribution model (e.g., gaussian, uniform, etc.) and construct an unbiased estimate of the distribution.
Consider a general case where the true conditional distribution is parameterized by P ( X miss | X obs , A )
=
f imp ( θ, X obs , A ) .
We consider the family of imputation models that are correctly-specified: { f imp (ˆ θ, X obs , A ) } ˆ θ ∈ Θ .
Similar to
Theorem 1, one can derive information lower bound for these correctly-specified imputation models.
The lower bound will not only include the conditional variance V ar ( X missij | M i = 1 , A i = a, X obs ) , but also the bias of the parameter estimate
ˆ θ .
A detailed analysis is provided in the Appendix A.1.
Fairness-awared Imputation GAN (FIGAN)
In Section 2, we have provided discussions on the imputation fairness risk of those correctly-specified imputation models.
In general, however, one may not have much information to construct a correctlyspecified parametric imputation model.
As an alternative solution, state-of-the-art deep learning models are adopted to construct non-parametric imputation models.
Following this idea, in this paper, we propose a fairness-aware imputation model based on generative adversarial networks (GAN).
The original GAN [Goodfellow et al., 2020] learns the underlying distribution of a given dataset and artificially generates fake data that are similar to the original data.
We proposed Fairness-aware Imputation GAN (FIGAN) in Algorithm 1, which consists of two neural networks: generator G and discriminator D .
Figure 2 shows the FIGAN’s framework.
At a high level, the generator G uses observed information of a sample, X obs , sensitive attribute A and random noise z to impute the missing values X miss .
The discriminator D learns to discriminate the imputed data ˆ X miss from the ground truth X miss .
To control the imputation fairness risk, a fairness regularization is imposed into the training objective function.
For the discriminator D , it is trained by minimizing the loss function
[Equation]
The generator G is trained by minimizing the loss function
[Equation]
in which λ acc and λ fair are tunable hyper-parameters.
The first term in L G forces the generator G to produce imputed values that can "fool" the discriminator D , and the second term forces to produce accurate imputations, the third term regularizes the model to have small imputation fairness risk.
Input: Complete cases (see definition in Figure 1)
X cc =
X obscc ∪ X misscc , sensitive attribute A cc , number of iteration EPOCHS.
: Fairness-aware Imputation GAN (i.e., G and D ).
for t in { 1 , . . .
, EPOCHS } do
(1) Sample the noise z ∼ N (0 , I ) .
(2) Impute the missing values via ˆ X misscc =
G ( X obscc , A cc , z ) .
(3) Forward the imputed values ˆ X miss cc and true missing values X miss cc into discriminator D .
(4) Update the discriminator using L D (3): θ ( t +1) D
(5) Update the generator using L G (4): θ ( t +1) G
← end for
Noticed that the loss function L D contains the ground truth value X miss , FIGAN will be trained on complete cases (Figure 1) in which the ground truth values X miss are known.
Specifically, for each complete case, we can generate the imputed values using fully observed features X obs and train
FIGAN using X miss and G ( X obs , A, z )) .
Empirical Results
In this section, we study the imputation performance of proposed FIGAN and 5 other existing methods on both synthetic and real datasets: MICE [Van Buuren and Oudshoorn, 1999], Misforest
[Stekhoven
[Figure]
and Bühlmann, 2012], SoftImpute [Mazumder et al., 2010], Sinkhorn [Muzellec et al., 2020] and
Gain [Yoon et al., 2018].
We illustrate that FIGAN is capable of achieving decent imputation accuracy while providing fairer imputation results.
4.1 Synthetic datasets
We generate the synthetic dataset via a time-series model.
In particular, X i +1 = 0 .
1 X i
+ ϵ
i + with ϵ i ∼ N (0 , 1) being independent white noise.
We set the sample size n = 1000 , number of features p = 25 .
We artificially generate missing values for 4 features, under three missing mechanisms: logit ( P ( M i = 1))
= 0 .
7 ; MAR, logit ( P ( M i = 1))
= 0 .
1 E empirical X obs ; MNAR, logit ( P ( M i = 1))
= 0 .
1 E empirical X miss .
The results are summarized into Table
[Table]
reported.
The results in Table 1 show that FIGAN has the smallest imputation fairness and smallest MSIE simultaneously.
This outperformance demonstrates FIGAN’s potential to provide accurate and fair imputations in practice.
4.2 Real datasets
To study the performance of FIGAN in practice, we conduct experiments on two real-world datasets,
COMPAS [Northpointe, 2010] and ADNI.
In both experiments, to assess the MSIE, we will artificially generate missing values, and compare the imputed values with the ground truth values.
More details of the datasets and experiment setup can be found in the Appendix A.2.
COMPAS recidivism dataset:
Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) [Northpointe, 2010] dataset contains records of defendants from Broward County from 2013 and 2014.
In our analysis, gender is treated as the sensitive attribute, and we only extract and use 12 non-categorical features in our experiments.
We generate missing values under three missing mechanisms: MCAR, logit ( P ( M i = 1))
= 0 .
7 ; MAR, logit ( P ( M i = 1))
= 3+0 .
4 E empirical X obs ; MNAR, logit ( P ( M i = 1)) =
− 2 E empirical X miss .
The results are summarized in Table 2.
From the table, we observe that FIGAN consistently has significantly lower imputation fairness risks, compared to all the other imputation methods.
In addition, FIGAN also achieves comparable, if not the best, MSIE in the experiment
[Table]
ADNI dataset:
Launched in 2003, the primary goal of the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database is to test whether it is possible to measure the progression of mild cognitive impairment and early Alzheimer’s disease.
The database contains patients’ positron emission tomography (PET), serial magnetic resonance imaging (MRI), other biological markers, and clinical and neuropsychological assessments.
The dataset adopted in this paper contains genomic information of 649 patients, who potentially have Alzheimer’s disease.
Among the 19k features in the original dataset, we only choose the first 1000 columns from the raw data in experiments.
We set race as the sensitive attribute and artificially generate missing values for the first 10 columns in the dataset.
The missing data are generated under three missing mechanisms: logit ( P ( M i = 1))
= 0 .
5 ; MAR, logit ( P ( M i = 1))
= 0 .
1 E empirical X obs ; MNAR, logit ( P ( M i = 1))
= 0 .
1 E empirical X miss .The
results are summarized in Table 2.
Similar to the experiments on the COMPAS dataset, from Table 2, we also observe that
FIGAN consistently has significantly lower imputation fairness risk and achieves comparable MSIE in the experiment.
Discussions
In this paper, we study the fairness associated with missing data imputation.
We first provide a theoretical analysis that suggests that there exists inherent imputation unfairness when using the ground truth conditional distribution of the missing data for imputation.
Furthermore, we propose
FIGAN, a GAN-based imputation model that empirically provides accurate and fair imputations.
We believe there is much room remaining for future study.
Firstly, there are widely used fairness notions based on causality, and investigations based on these causal fairness notions could provide unique insights as well.
Secondly, other than GAN, there have been other generative models such as autoencoders and diffusion models that achieve decent performance in synthetic data generation.
We believe that fair imputation models based on these generative models can be an exciting direction for future study.
Thirdly, theoretical analysis on the fairness of FIGAN could also be insightful, we expect it to help better understand the intuitions and strategies of improving fairness during missing data imputation.
A Appendix A.1 Theoretical analysis on correctly-specified imputation model
We begin by providing the proof of Theorem 1.
Proof.
Taking the expectation of the mean squared imputation error, we have that
[Equation]
where the second equation holds since all the features in X miss are missing simultaneously.
By law of total expectations,
[Equation]
Since we only have the information about ( X obs , A i , M i ) during the imputation [Equation] the imputed value ˆ X missij is independent of the true value X missij , conditioned on ( X obs , A i , M i ) .
Moreover, since we are using the true conditional distribution P ( X miss | X obs , A ) , ˆ X missij has the same conditional distribution as X missij .
Hence we have that
[Equation]
with σ 2 a,j = V ar ( X missij | M i = 1 ,
A i = a, X obs ) .
Finally, by triangle inequality,
[-E [ IFR ( f imp )] = E | MSIE A =
 ( f imp )-][Equation]
[-− MSIE A =1 ( f imp ) | ≥ | E [ MSIE A =
-]
combining equation (5) and (6) into the above expression completes the proof.
Moving one step further, we provide an analysis of the imputation fairness for correctly specified imputation models.
Theorem 2.
For a dataset that has single-pattern missing data (i.e., m missij ≡ M i ), assume a correctly specified imputation model f imp (ˆ θ, X obs , A ) is adopted for imputation, then imputation fairness risk has the lower bound:
[Equation]
where K is the number of features that has missing values (i.e., the number of columns in X miss ), and
[Equation]
For consistent estimate ˆ θ , we have that ˜ σ 2 a,j → σ 2 a,j as n → ∞ .
Proof.
From the proof of Theorem [Equation] we know that
[Equation]
In addition, the imputed value ˆ X missij is independent of the true value X missij , conditioned on ( X obs , A i , M i ) .
We have that
[Equation]
E (( ˆ X missij − X missij ) 2 | M i = 1 , A i = a, X obs ) = E X obs [ E (( ˆ X missij ) 2 | M i = 1 , A i = a, X obs ) + E (( X missij ) 2 | M i = 1 , A i = a, X obs )
− 2 E ( ˆ X missij | M i = 1 ,
A i = a, X obs ) E
( X missij | M i = 1 , A i = a, X obs )]
For correctly-specified imputation model f imp (ˆ θ, X obs , A ) , applying the Taylor expansion yields:
[Equation]
and
[Equation]
combine together above three equations, we have that
[Equation]
Finally, by triangle inequality,
[Equation]
combining equation (5) and (7) into the above expression completes the proof.
From Theorem 2, we know that the bias of the parameter estimator ˆ θ will influence the lower bound of the imputation fairness.
A.2 Experiment details
In this section, we provide details of the experiments in Section 4.
In our experiments,
FIGAN is implemented in Pytorch (version 3.6.9).
MICE and MissForest are run using ( sklearn.impute.
IterativeImputer ), Sinkhorn (Optimal Transport) is run using the original author’s code, SoftImpute and Gain are implemented in Python.
In terms of the model architecture, three fully-connected layers are adopted with Tanh activations in the generator G , three fully-connected layers layers are adopted with ReLU activations in the dsicriminator D .
FIGAN is trained with epochs = 50 , batch size = 16 , learning rate = 0 .
001 , λ acc
= 1 .
For the synthetic data experiments, we choose the features at j -th columns with j ∈ [2 , 4 , 7 , 9 , 12 , 14 , 17 , 19] to have missing data, simultaneously.
We generate the sensitive attribute A by A i = 2 ∗ 1 { mean ( X i ) ≥ 0 } − 1 .
For the COMPAS dataset, we take all the 12 non-categorical features, and generate missing values on age, prior crime counts and other 4 features.
After preprocessing, there are 5479 male and 1326 female recorded.
For ADNI dataset, there are 643 white andblack patients.
The influence of such significant imbalance is reflected by the values of IFR in Table 2.