H OW TO DEAL WITH MISSING DATA IN SUPERVISED DEEP LEARNING ?
Niels Bruun Ipsen ∗ nbip@dtu.dk
Pierre-Alexandre Mattei †‡ pierre-alexandre.mattei@inria.fr
Jes Frellsen ∗‡ jefr@dtu.dk
A BSTRACT
The issue of missing data in supervised learning has been largely overlooked
, especially in the deep learning community
We investigate strategies to adapt neural architectures for handling missing values
Here
, we focus on regression and clas siﬁcation problems where the features are assumed to be missing at random
Of particular interest are schemes that allow reusing as is a neural discriminative ar
- chitecture.
To address supervised deep learning with missing values, we propose to marginalize over missing values in a joint model of covariates and outcomes
Thereby, we leverage both the ﬂexibility of deep generative models to describe the distribution of the covariates and the power of purely discriminative models to make predictions.
More precisely
, a deep latent variable model can be learned jointly with the discriminative model, using importance-weighted variational inference, essentially using importance sampling to mimick averaging over multiple imputations.
In low-capacity regimes or when the discriminative model has a strong inductive bias, we ﬁnd that our hybrid generative/discriminative approach generally outperforms single imputations methods
I NTRODUCTION
[Equation]
[Figure]
[Equation]
[Figure]
γ
N
Missing data affect data analysis across a wide range of domains and the sources of missing values span an equally wide range.
Recently, deep latent variable models (DLVMs, Kingma & Welling, 2014; Rezende et al., 2014) have been applied to missing data problems in an unsupervised setting (e.g. Rezende et al., 2014;
Nazabal et al., 2020; Ma et al., 2018; 2019;
Ivanov et al., 2019; Mattei & Frellsen, 2018; 2019; Yoon et al., 2018; Li et al., 2019; Ipsen et al., 2021; Ghalebikesabi et al., 2021), while the supervised setting has not seen the same attention.
The progress in the unsupervised setting is focused on inference and imputation and can therefore be useful as an imputation step before learning a discriminative model.
Traditionally the focus has also been on inference and imputation either as a goal in itself or before supervised learning on the (possibly multiple) imputations and observed data (Rubin, 1976; 1996; Little & Rubin, 2019).
Supervised learning in the presence of missing values has different goals and pose different challenges than inference and imputation (Josse et al., 2019; Le Morvan et al., 2020a; 2021).
Here the overall aim is to minimize an expected prediction error.
However, optimal single imputation does not necessarily lead to optimal prediction in terms of minimizing a prediction error (Josse et al., 2019).
One challenge is that predictions based on inputs with missing values can be ambiguous, that is, the conditional distribution of the missing values given the observed may be multimodal and the optimal prediction may change with the mode.
With single imputation, the conditional distribution over the missing data is discarded, and the optimal single imputation can no longer reﬂect this ambiguity, see ﬁgure 2.
Even the optimal single imputation leads to biased parameter estimates compared to the fully observed case (Bertsimas et al., 2021), and may in some cases lie outside the distribution of the data.
Another challenge is that the number of missing value patterns grows combinatorially with the number of features p , requiring 2 p submodels to ﬁt the Bayes predictor in the linear case (Le Morvan et al., 2020a).
In multiple imputation (Rubin, 1996), several imputations are drawn from the posterior predictive distribution of the missing values, reﬂecting the full conditional distribution of the missing values and thus the uncertainty about what is missing.
This allows for uncertainty estimates in downstream tasks such as prediction.
This in turn requires ﬁtting as many discriminative models as the number of imputations.
1. C ONTRIBUTIONS
In order to address supervised deep learning with missing values we develop the supervised missing data importance-weighted autoencoder (supMIWAE) bound , based on the approximate maximum likehood techniques used by Burda et al.
and Mattei & Frellsen (2019).
This is a scalable approach to marginalizing over missing values in a joint model of covariates and outcomes
[Equation]
The covariate model p θ ( x obs ,
[Equation]
miss ) =
(cid:82) p ( x obs , x miss | z ) p ( z )
d z is a DLVM in this work, but can be any probabilistic model imposing a joint density over covariates.
The outcome model p φ ( y | x obs , x miss ) is any neural discriminative architecture that would have been used in the complete-data case, parameterizing a density over the outcomes.
The graphical model is shown in ﬁgure 1 along with its computational structure.
Once the joint model has been trained using the supMIWAE bound, the conditional distribution of the outcome given the observed parts of the input can be found as
[Equation]
This is approximated using importance sampling techniques, averaging predictions over multiple importance samples (akin to multiple imputations, see appendix C for a deeper discussion) from the generative model.
The model can be trained end-to-end or a pretrained generative model can be coupled with a discriminative model.
Having a joint model allows for supervised imputations, where available labels can help guide the imputations from the generative model by adjusting the importance weights accordingly.
Joint models over covariates and labels have previously been used to marginalize over missing values in the covariates, using Gaussian mixture models either as the model over the covariates or directly as the joint model over covariates and outcomes (Ghahramani & Jordan, 1995; Ghahramani & Hinton, 1996; Ahmad & Tresp, 1993;
Tresp et al., 1994; 1995; ´
Smieja et al., 2018).
Our approach shows how to use more ﬂexible generative models, such as DLVMs, provides an efﬁcient optimization procedure and allows for keeping any deep discriminative architecture unchanged.
B ACKGROUND AND NOTATION
We deﬁne the random variable x
= ( x 1 , . .
, x p ) ∈ X which takes values in a p -dimensional feature space X
= X 1 × · · · × X p .
There is a corresponding (possibly vector valued) response variable y ∈ Y .
A missing process obscures parts of x resulting in the mask random variable s ∈ { 0 , 1 } p where
[Equation]
[Figure]
We let obs ( s ) denote the indices of the non-zero entries of s and miss ( s ) denote the indices of the zero-entries of s , such that x obs ( s ) is the set of observed elements of x , and x miss ( s ) is the set of missing elements of x .
For simplicity we will omit the s and write x obs , x miss respectively, whenever the context is clear.
Following (Yoon et al., 2018; Le Morvan et al., 2020b; Ghalebikesabi et al., 2021)
we deﬁne the incomplete random variable ˜ x
= (˜ x 1 , . .
. , ˜ x p ) ∈ ˜
X which takes values in ˜ X
=
( X 1 ∪{ na } ) ×· · ·× ( X p ∪ { na } ) where missing values are represented by the symbol na , such that na · x j = na and na · 0 = 0 .
We typically only have access to ˜ x , where
[Equation]
and (cid:12) is the Hadamard product.
Finally, we are given n i.i.d.
copies of the random variables ˜ x and y which we collect in a dataset D =
{ ˜ x i , y i } ni =1 , or alternatively D = (cid:8) x obs i , y i (cid:9) n i =1 .
We make the additional assumption that the data are missing at random (MAR, see e.g. Seaman et al., 2013; Little & Rubin, 2019).
Speciﬁcally, this means that we assume that s and x miss are independent given x obs .
This assumption allows to avoid to explicitly model the missing mechanism.
Our approach could be extended beyond MAR by using a generative model ﬁt for this purpose, like the not-MIWAE of Ipsen et al.
(2021), or the deep pattern-set mixture of Ghalebikesabi et al.
2. C HALLENGES WHEN TRAINING DISRIMINATIVE MODELS WITH MISSING DATA
Our predictive model will be deﬁned through a mapping f φ : X → H used to parameterize a conditional distribution
[Equation]
Here (Ψ( ·| η ))
η ∈ H is a parametric family of distributions over the outcome space Y , such as a Gaussian distribution in regression or a categorical distribution in classiﬁcation.
Under the maximumlikelihood framework (or equivalently the logarithmic scoring rule), an optimal mapping f ∗ φ within a class F = ( f φ ) φ ∈ Φ parameterized by φ ∈ Φ is deﬁned as
[Equation]
[Equation]
where p ∗ ( x , y ) is the true data generating distribution.
When the data are complete, this is typically approximated by maximizing the log likelihood of the parameters φ given the data D train , (cid:96)
[Equation]
log p
[Equation]
|
[Equation]
When the covariates have missing values the log-likelihood cannot be maximized directly as the likelihood is not deﬁned since p φ ( y | x ) depends on the full input vector.
The simplest approach is instead to learn separate mappings for each missing pattern.
This reduces the amount of data available for the training of each mapping and does not scale well with the dimensionality of the input as in general 2 p networks need to be trained.
An often used approach is instead to impute the missing values using a single imputation (for instance, using E p θ ( x miss | x obs ) [ x miss ] or an approximation of it), and then map the concatenation of observed and imputed values to approximate the conditional distribution
[Equation]
Instead of E p θ ( x miss | x obs ) [ x miss ] , which is the optimal imputation under the mean squared error, one could use any kind of imputation, for instance using a constant, or the mean.
This general approach is adequately called impute-then-regress by Bertsimas et al.
(2021) and Le
Morvan et al.
While it will be Bayes-consistent given a powerful enough classiﬁer (Le Morvan et al., 2021), it leads to biased parameter estimates compared to having complete data (Bertsimas et al., 2021).
This is illustrated in ﬁgure 2 where the optimal (under the mean squared error) single imputations are placed entirely within one of the classes, requiring a biased decision surface to properly reﬂect the class label probabilities for the records with missing values.
From a probabilistic perspective, computing p φ ( y | x obs ) requires marginalizing over the missing features as in equation (1), that is
[Equation]
Notice the difference to single imputation since, in general
[Equation]
except in some pathological cases (e.g. when p φ is linear or p θ is a Dirac distribution).
Therefore, if the discriminative model is nonlinear and the generative model is complex enough, then using single imputation will give a very different result than marginalising over the missing features.
This is exempliﬁed in ﬁgure 2, where a classiﬁer trained using the optimal imputation ﬁnds a solution that is very different to what would be found by one trained without missing data.
Our technique, marginalizing the missing values, allows to recover the same decision surface as a classiﬁer trained without missing data.
2. O UR JOINT MODEL AND ITS SUP MIWAE VARIATIONAL BOUND
As mentioned in the introduction, we posit a joint model over covariates and outcome, that will make use of a latent variable z :
[Equation]
Under the MAR assumption, the relevant substitute of the likelihood is the likelihood of the observed data
[Equation]
The integral in equation (12) is in all but the simplest cases analytically intractable and direct maximum likelihood methods for learning the parameters ( θ , φ ) cannot be used.
In order to learn the parameters of the joint model, we turn to amortized importance weighted variational inference , where a lower bound is maximized instead of the log likelihood itself (Burda et al., 2016;
Domke & Sheldon, 2018).
It is based on the fact that unbiased estimates of a likelihood can be turned into lower bounds of the log-likelihood.
Indeed, if R ( x obs , y ) is a random variable such that E (cid:2)
R ( x obs , y ) (cid:3) =
p θ , φ ( x obs , y ) , then E (cid:2) log R ( x obs , y ) (cid:3) ≤ log E (cid:2) R ( x obs , y ) (cid:3)
= log p θ , φ ( x obs , y ) , which means that E (cid:2) log R ( x obs , y ) (cid:3) is a lower bound of the log likelihood.
A simple way to ﬁnd a suitable R ( x obs , y ) is using importance sampling from a variational distribution over z .
More speciﬁcally, let
[Equation]
where q γ ( z | x obs ) is the variational distribution (learnable proposal) and ( x miss k , z k ) k ∈{ 1 ,...,K } are i.i.d.
samples from p θ ( x miss | z ) q γ ( z | x obs ) .
Then we have that E ( x miss k , z k ) (cid:2) R K ( x obs , y ) (cid:3)
= p θ , φ ( x obs , y ) and E ( x miss k , z k ) (cid:2) log R K ( x obs , y ) (cid:3) is a lower bound on the log likelihood.
The variational distribution can be parameterized via a neural network with weights γ ∈ Γ , similarly to
Mattei & Frellsen (2019), Nazabal et al.
(2020), and Ipsen et al.
Now that we have a way to bound each per-datum log-likelihood, we can deﬁne our supMIWAE variational bound L K ( θ , φ , γ ) that is a lower-bound of the log-likelihood of the observed data
[Equation]
The lower bound L K ( θ , φ , γ ) is a Monte Carlo objective that can be maximized instead of the log likelihood itself, using techniques from stochastic optimization (see e.g. Mohamed et al., 2020).
As detailed in appendix D, it beneﬁts from the usual theoretical advantages of importance-weighted variational bounds:
L K ( θ , φ , γ ) converges monotonically to the loglikelihood (cid:80) ni
=1 log p φ , θ ( y i , x obs i ) at speed 1 /K .
This means that, the larger the K , the closer our objective will be to the true likelihood.
Note that the contribution of any fully observed data point ( x , y ) will be simply
[Equation]
which is just the sum of the discriminative likelihood and the standard IWAE bound of Burda et al.
An interesting property of our bound is that the discriminative and generative parts can be trained jointly with a single and statistically sound objective.
However, if we were to encounter a data set without missing data, equation (15) means that the training of the two parts of the models would be decoupled: the discriminative model would be the same as a standardly trained discriminative model, and the generative model would be the same as a standard IWAE-trained DLVM.
2. P REDICTION
Once the model has been jointly trained using the supMIWAE bound, it is possible to use it to do prediction with missing data.
Indeed self-normalized importance sampling can be used to approximate the conditional
[Equation]
(16) where
[Equation]
(17) and ( x miss k , z k ) k ∈{ 1 ,...,K } are i.i.d.
samples from p θ ( x miss | z ) q γ ( z | x obs ) .
Again, this technique mimicks multiple imputation: several imputations are generated using the generative model, are then fed to the classiﬁer, and the predictions of each imputation are averaged.
R ELATED WORK
Marginalizing over missing covariates Different approaches have previously been taken to marginalize over missing values in joint models over features and labels.
In a series of publications (Ahmad & Tresp, 1993; Tresp et al., 1994; 1995), a closed-form solution using Gaussian Basis
Function networks was used to approximate this marginalization, while keeping the neural discriminative model ﬁxed.
Ghahramani & Jordan (1995) took a different approach, modeling the joint distribution p ( y , x ) directly as a mixture model, obtaining p ( y | x obs ) by conditioning on observed quantities in the joint distribution, thus removing the explicit discriminative neural architecture.
In the context of kernel methods, Dick et al.
(2008) proposed to learn a distribution over missing values by minimising the regularized empirical risk.
´
Smieja et al.
(2018) trained a Gaussian mixture model (GMM) and discriminative model jointly, and in place of any missing values the activation of the corresponding input neuron was set to the average activation over the GMM conditioned on observed values.
Consistency of single imputation A review of approaches to handling missing data in (non-deep)
supervised learning was given by Josse et al.
Here it was shown that under some assumptions about the capacity of the learner, constant imputation is asymptotically consistent in the supervised setting.
Le Morvan et al.
(2020b) investigated the case of a linear predictor on covariates with missing data, showing that in the presence of missing, the optimal predictor may not be linear and how constant imputation of each feature can be optimized with regards to the model loss.
The linear case was further investigated in (Le Morvan et al., 2020a), deriving the analytical form of the optimal predictor and proposing NeuMiss networks to approximate this.
Bertsimas et al.
(2021) analyzed the impute-then-regress situation and noted that while this paradigm is widely used in practice it incurs a bias in the parameter estimates, but mean imputation is still asymptotically consistent.
Le
Morvan et al.
(2021) showed that the impute-then-regress approach is asymptotically Bayes optimal given a learner with large capacity and that this holds for all missing mechanisms, even missing not at random (Rubin, 1976).
However, the choice of imputation function affects the difﬁculty of the subsequent regression task, when not in the asymptotic regime.
As a consequence they proposed to learn imputation and regression jointly, where the missing values are handled by a NeuMiss network.
Other approaches for missing covariates Yi et al.
(2019) tackled the issue of sparsity, and specifically large variations in sparsity, by introducing sparsity normalization.
This addresses the issue of model output covarying with the sparsity level in the input.
Ma et al.
(2018; 2019) used a permutation invariant setup to avoid imputing missing data in the input of a variational autoencoder.
This approach can be readily extended to the supervised setting, using the permutation invariant setup as a modiﬁed input layer as we show in appendix A.3.
Some tree approaches to regression/classiﬁcation can use observations with missing values directly (see e.g. Twala et al., 2008; Chen & Guestrin, 2016; Josse et al., 2019; G ´ omez-M ´ endez & Joly, 2021).
Sparse coding was used by Caiafa et al.
(2020) to train a dictionary and a classiﬁer simultaneously.
DLVMs and missing values
Deep latent variable models have been applied to missing data problems in the unsupervised setting, focused on inference, single and multiple imputations.
Early works assumed that a DLVM had previously been trained on complete data (Rezende et al., 2014; Mattei & Frellsen, 2018), but a variety of techniques to handle incomplete training sets was then proposed, both in M(C)AR (Yoon et al., 2018;
Ma et al., 2018; 2019; Ivanov et al., 2019; Mattei & Frellsen, 2019;
Li et al., 2019) and MNAR (Ipsen et al., 2021; Ghalebikesabi et al., 2021) cases.
In (Li &
Marlin, 2020) irregularly sampled time-series are approached as a missing data problem and they propose VAE and GAN based models for handling this.
The proposed models can also be used for supervised learning by training a classiﬁer on the latent representations, either by training the model jointly or using a pretrained encoder.
Without missing data, the idea of jointly training a VAE and a discriminative model, has been explored by Ji et al.
(2020) or Joy et al.
E XPERIMENTS
We now evaluate discriminative models trained using the supMIWAE bound on a range of supervised learning tasks.
Throughout the experiments the generative part of the model is pretrained using the MIWAE bound of Mattei & Frellsen (2019).
The pretrained DLVM can be used to draw single imputations, referred to as MIWAE single imputation, or it can be used as the generative part in the supMIWAE computational structure.
Here, the generative part of the model is kept ﬁxed while updating the discriminative part of the model according to the supMIWAE bound.
As the DLVM in
MIWAE single imputations and the supMIWAE are similar, any difference in performance between the two is attributed to their different strategies for handling missing values.
Other strategies for handling missing data, used for comparison in this work, are brieﬂy described in appendix A. Missing values are introduced in the training, validation and test sets according to the missing mechanism used in the given experiment.
Predictive performance is measured on a test-set with missing values.
In all experiments, there is a computational overhead compared to single imputation of training the supMIWAE that scales linear with the number of importance samples K , c.f. equation (13).
4. 2D CLASSIFICATION
A qualitative analysis of the bias in the learnt parameters of the discriminative model, due to single imputation, is set up on simple 2D datasets, see ﬁgure 8 of appendix
E. The simplicity allows us
[Figure]
to directly inspect the impact of different imputation strategies on the learnt decision surface and compare this to the marginalization procedure used in the supMIWAE.
All datasets consist of a training set with 3k records and validation and test sets with 1k records.
An MCAR missing process is introduced in the horizontal coordinate, where each element becomes missing with probability m = 0 .
Further training details are in appendix
E. The datasets are such that there will be inherent ambiguity about the class label for some of the records with missing values.
For example in the Burger dataset, all records with a missing value will have p ( y = 1 | x obs ) = 2 / 3 .
In ﬁgure 3 imputations from different imputation models are shown along with the corresponding learnt decision surface.
Partially observed examples are shown in the margins of the plots as sticks, colored by their class label.
All methods use the same discriminative architecture, only the imputation strategy differs.
For the supMIWAE, multiple imputations are shown for three partially observed records, alongside their kernel density.
These multiple imputations reﬂect the appearance of importance samples used during training, for these three partially observed records.
Decision surface artifacts The single imputation methods introduce artifacts into the learnt decision surface.
As the discriminative model tries to reﬂect the proper conditional probabilities p ( y | x obs ) at the location of the imputed values, the ambiguity about the class label needs to be reﬂected in the predictions.
Notice how the MIWAE produces nearly optimal imputations in terms of minimizing a mean-square-error and how this translates into decision surface artifacts.
These artifacts are related to the notion of imputation manifolds used by Le Morvan et al.
(2021) to prove their main theorem on the Bayes consitency of impute-then-regress procedures.
The remaining datasets are shown in ﬁgures 10–13 of appendix E, where also the permutation-invariance setup (appendix A.3) and histogram-based gradient boosting are included.
Like the supMIWAE, these two approaches do not rely on explicit imputations.
As the supMIWAE can draw multiple importance samples, instead of one single imputation, the conditional probabilities p ( y | x obs ) can be properly reﬂected, when averaging over the importance samples.
This avoids the single imputation artefacts.
Predictive performance Do the artifacts introduced by single imputation methods, compared to marginalizing over the missing values, translate into differences in predictive performance?
The predictive performance corresponding to the decision surfaces in ﬁgure 3 are shown in table 1 in appendix E. Both the test-set accuracies and log likelihoods are more or less similar across the different strategies for handling missing values.
This aligns with the results by Josse et al.
; Bertsimas et al.
Le Morvan et al.
(2021): the impute-then-regress procedures with powerful learners are consistent and Bayes optimal, i.e. when the learner has high capacity any single imputation can essentially be undone.
While single-imputation methods can lead to Bayes-optimal learners, Le
Morvan et al.
(2021) address that the difﬁculty of the learning problem depends on the choice of imputation function.
This is seen in ﬁgure 4; while the supMIWAE and 0-imputation eventually ends up with similar performance the learning problem appears easier for the supMIWAE.
For 0imputation the learner has to introduce high-frequency artifacts to the decision surface in order to get the conditional probabilities p ( y | x obs ) calibrated, but in the case of a high capacity learner this can eventually be done.
Limited capacity learner We now turn to a situation where the capacity of the learner is limited in order to assess how single imputation methods fare against marginalizing over missing values.
We start by noting that the decision surface for the Burger dataset, in the complete-data case, can be learnt using two sigmoids.
When using MSE-optimal single imputations in the missing-values case this changes, as now 4 sigmoids are needed to make sure the imputations are well calibrated.
This illustrates that the capacity of the learner determines whether single imputations can be undone.
In ﬁgure 5 we explore how the capacity of the learner, in terms of the number of hidden units, affects the predictive performance, depending on the imputation strategy.
More capacity experiments are found in ﬁgure 14.
4. I MAGE C LASSIFICATION
When the learning problems become harder, strong inductive biases are sometimes used to help learning.
This is typically the case with image classiﬁcation where convolutional neural networks (CNNs) are often used to introduce translational equivariance.
The previous section showed that the strategy for handling missing data affects predictive performance when the learner is not in the high capacity regime.
This is further investigated in terms of classiﬁcation accuracy in image experiments with different MCAR missing processes, such as observed squares, missing squares and random dropout, see ﬁgures 15–17.
The discriminator is a convolutional neural network, see appendix F.1, thus introducing an inductive bias in the learner that has proved useful in image classiﬁcation in the complete-data case.
While CNNs are ﬁt for images, it should be noted that they are not universal approximators, and may consequently have a hard time undoing single imputations.
We apply the supMIWAE and single imputations methods to the MNIST dataset (LeCun et al., 1998) and the fashion MNIST dataset (Xiao et al., 2017) with three different MCAR missing mechanisms: 1) observed squares, where a 12 × 12 randomly placed square is observed, 2) missing squares, where a 12 × 12 randomly placed square is missing, and 3) dropout, where each pixel is missing with some constant probability m , over a range of missing rates.
In practice, we see that this setting where there is a strong inductive bias is similar to our previous low-capacity experiments: for both MNIST and Fashion MNIST, the supMIWAE is signiﬁcantly more accurate than single-imputation competitors, as seen in ﬁgure 6.
Additional experiments, including experiments on natural images (SVHN) and visualizations of multiple imputations, are available in appendix F.
4. R EGRESSION
We now turn to regression in smaller and lower dimensional datasets from the UCI database (Dua & Graff, 2017).
An experimental setup modiﬁed from Hern ´ andez-Lobato & Adams (2015) and
Skafte et al.
(2019) is used here.
Predictive performance in terms of root-mean-square-error is presented over a range of missing rates.
Results are seen in ﬁgure 7, and more results can be found in ﬁgure 20 of appendix G.
There is no clear winner here, although it appears that gradient boosting generally performs quite well, and that accurate single imputations (such as MIWAE’s or
MissForest’s) generally outperform inaccurate ones (such as zero imputation).
In general, our joint model performs as well as feeding accurate single imputations.
C ONCLUSION
In the context of supervised deep learning with missing values, single imputation methods are an often used tool.
While this approach is asymptotically consistent it incurs a bias in the learnt parameters.
We have introduced a scalable approach to marginalizing over missing values using deep generative models, as a probabilistic alternative to single imputation.
Besides limiting single imputation artifacts our results indicate that there are two cases where using a generative model can be quite valuable for supervised learning with missing values: when the classiﬁer is not very ﬂexible, or has a strong inductive bias.
Moreover, leveraging such a generative model allows us to be able to use untouched any discriminative architecture (for instance, any pretrained one).
R EPRODUCIBILITY S TATEMENT
Code for reproducing paper experiments is available at https://github.com/nbip/ supMIWAE .
A CKNOWLEDGEMENTS
This work has been supported by the French government, through the 3IA Cˆote d’Azur Investments in the Future project managed by the National Research Agency (ANR) with the reference number ANR-19-P3IA-0002.
Furthermore, it was supported by the Novo Nordisk Foundation (NNF20OC0062606 and NNF20OC0065611) and the Independent Research Fund Denmark (913100082B)
A T RAINING DEEP SUPERVISED MODELS WITH MISSING DATA
In the following we group different strategies for handling missing data in supervised deep learning.
A. C ONSTANT IMPUTATION
We denote imputation methods where there is a single constant for each feature which replaces missing data as constant imputation .
We deﬁne an imputation function ι in general such that ι (˜ x ) ∈
X and ι (˜ x ) obs = x obs .
A simple version of constant imputation is 0-imputation, which has the intuitive appeal that the activation from the input node is zeroed out (absent) and no gradient updates are made for the weights for that feature,
[Equation]
[Equation]
In the unsupervised setting, constant imputation biases marginal and joint distributions, but Josse et al.
; Le Morvan et al.
; Bertsimas et al.
(2021) have shown that mean imputation can be consistent in the supervised setting.
Furthermore, Le Morvan et al. (2020b); Tresp et al.
(1994) noted that the constants can be optimized with respect to the prediction loss.
Learnable parameters λ ∈ X can be inserted in place of the missing data, so that
[Equation]
We denote this learnable imputation .
A. R EGRESSIVE /M ODEL B ASED SINGLE IMPUTATION
In contrast to constant imputation, methods such as MICE (Buuren & Groothuis-Oudshoorn, 2010),
MissForest (Stekhoven & B ¨ uhlmann, 2012), PPCA (Tipping & Bishop, 1999; Roweis, 1998) or ﬂexible generative models (Mattei & Frellsen, 2019; Yoon et al., 2018) provides imputation of missing features conditional on observed features p ( x miss | x obs ) ,
[Equation]
In MICE, this conditional distribution comes from an implicit joint model over the covariates by regressing each covariate with missing values on all the others, in a round-robin fashion.
MissForest is somewhat similar to MICE, using random forests as the regressors.
Probabilistic PCA deﬁnes an explicit joint model over all covariates and single imputations of missing data can be obtained by conditioning on observed covariates.
Here, MICE is a proper Bayesian multiple imputation method, while PPCA and MissForest draws multiple imputations without regard to parameter uncertainties.
The recent advances in imputation of missing data using deep generative models can also be used here to the extent that they impose a joint distribution over covariates.
A. P ERMUTATION INVARIANCE
Ma et al.
(2018; 2019) used the permutation invariant setup (Zaheer et al., 2017; Qi et al., 2017) to handle missing data in the context of variational autoencoders.
For each feature j ∈ obs ( s ) , there is an embedding e j ∈ R U , which in general can hold any auxiliary information, but in our case will be a learnable embedding.
Each element of the embedding is multiplied with the feature value t j = e j · x j and fed to a neural network h ( · ) , mapping inputs from R U to R M where U is the embedding dimension and M is the dimension of the code-space being mapped into.
This permutation invariant setup ﬁnally aggregates all the neural network outputs by a summation to get one ﬁxed-length vector, that can be fed into a discriminative network,
[Equation]
Note that in this case ι PI ( x obs ) ∈ R M , while in the other imputation functions ι ( ˜ x ) ∈ X .
A. T REE METHODS
Decision trees can in some cases handle missing data directly, without the need for imputation.
There are different approaches to handle missing values in tree based methods, see (Josse et al., 2019) for a review.
In our experiments we use histogram-based gradient boosting (Friedman, 2001), as implemented in scikit-learn (Pedregosa et al., 2011).
This approach uses the Block Propagation method for the missing data.
While a discriminative model using MLPs has a ﬁxed capacity, gradient boosting methods can adapt its capacity in terms of tree depth based on a validation set error.
B G RADIENTS
The log likelihood, when integrating over missing input features, is speciﬁed as
[Equation]
22) where
[Equation]
Both Tresp et al.
and Ghahramani & Jordan (1995) analyzed the gradient of the log likelihood with respect to the discriminative network parameters,
[Equation] [Equation]
Here we see that the gradient of the discriminative network parameters, due to the classiﬁcation/regression loss, is a weighted sum over all possible missing values, where the missing values are weighted according to their density in the data model, conditional on the observed data.
The gradient with respect to the θ parameters depends to some extend on the φ parameters.
[Equation]
There are contributions to the gradient from both the generative part of the model and the discriminative part.
This means that, if needed, the discriminative model can affect the generative model, as is the case in (Ipsen et al., 2021).
The gradient of the expectation can be approached using the reparameterization trick (Kingma & Welling, 2014; Rezende et al., 2014).
C R ELATION TO M ULTIPLE I MPUTATION
The gold standard for imputation is multiple imputation (Rubin, 1996).
This requires learning a model of the observed covariates x obs with parameters θ .
K sets of imputations are then drawn from
the posterior predictive distribution
[Equation]
where p ( x miss | x obs )
= (cid:82) p ( x miss | θ )
p ( θ | x obs )
d θ if the model is Bayesian (see e.g. Gelman et al., 2013, chapter 18.2).
Several frequentist multiple imputation methods also exist, where the uncertainty with respect to the imputation model parameters is disregarded.
In that case, the Bayesian posterior predictive is replaced by a conditional distribution p ( x miss | x obs ) =
p ˆ θ ( x miss | x obs ) , where ˆ θ is any point estimate of θ .
In the case of regression/classiﬁcation, if we roughly follow the notations of our paper, the analysis proceeds as follows
• For each set of imputed values ( x miss ik ) i ≤ n,k ≤ K , use a learning algorithm to estimate the discriminative parameters φ 1 , . .
. , φ K :
[Equation]
• Obtain an average estimate of φ using
[Equation]
and a variance estimate can similarly be obtained as described in (Gelman et al., 2013; Rubin, 1996).
In the limit of inﬁnite imputations, we can write this as the conditional expectation
[Equation]
Instead of ﬁtting K conditional models to K imputed datasets, our approach (when θ is ﬁxed) is, in comparison, to directly maximize the expectation of the conditional distribution with respect to the missing data
[Equation]
Also note that we have no need for several ﬁxed sets of imputations, at each gradient step a new set of samples are drawn from the imputation model, x miss ∼ p θ ( x miss | x obs ) .
The number of imputations can be set dynamically, using ∼ 5–50 during training and ∼ 10k for testing.
Our approach, using
DLVMs as imputation models, is not Bayesian, but any available proper Bayesian imputation model could be used in place of the DLVMs.
In practice, an important difference between traditional multiple imputation and our approach, is that multiple imputation uses a uniform weight 1 /K for each imputation, while we use importance weights, that will treat each imputation based on its likelihood.
D T HEORETICAL PROPERTIES OF THE SUP MIWAE BOUND
We will brieﬂy see how the supMIWAE bound has the same theoretical properties as IWAE bounds.
We follow Ipsen et al.
(2021, appendix D), who were following Burda et al.
(2016) and Domke &
Sheldon (2018).
Recall the deﬁnition of the bound: we have
[Equation] [Equation]
[Figure]
and
[Equation]
The i th term of the sum can be seen as a regular IWAE bound with observation ( x obs , y ) , latent variable ( z , x miss ) with prior p θ ( x miss | z ) p θ ( z ) , and variational distribution p θ ( x miss | z ) q γ
( z | x obs ) .
Therefore, Theorem 1 from Burda et al.
(2016) can be applied, and leads to the monotonicity of the bound:
L 1 ( θ , φ , γ ) ≤ . . .
≤ L K ( θ, φ, γ ) .
To show that the bound additionally converges to the log-likelihood when K −→ ∞ , we use Theorem 3 of Domke & Sheldon (2018) for all n terms of the sum that constitutes the bound.
Theorem.
Assuming that, for all i ∈ { 1 , ..., n } , •
there exists α i > 0 such that E (cid:2) | R 1 ( x obs i , y i ) − p θ
φ ( x obs i
[Equation]
y i
[Equation]
|
+ α i (cid:3) <
[Equation]
• lim sup K −→∞ E (cid:2) 1 /R K (cid:3) < ∞ ,
the supMIWAE bound converges to the true joint likelihood at rate 1 /K :
[Equation]
E 2D C LASSIFICATION
The datasets used in the 2D classiﬁcation experiments are shown in ﬁgure 8.
The Half-moons and
Circles datasets are from scikit-learn (Pedregosa et al., 2011).
The Pin-wheel dataset is modiﬁed from Johnson et al.
The Burger dataset, we have designed to illustrate the case where optimal single imputation requires severe changes in the decision surface.
The discriminative model is an MLP with 3 hidden layers with 50 hidden units each.
The ﬁnal layer parameterizes a categorical distribution over the dataset classes.
The generative model used for the MIWAE single imputations and the supMIWAE consists of an encoder and decoder with 3 hidden layers containing 50 hidden units each.
The encoder parameterizes an isotropic
Gaussian distribution over a latent space with dimension p , same as the feature space.
The decoder parameterizes an isotropic Gaussian distribution over the feature space.
In ﬁgure 9 samples from the generative model are shown, that is, samples from the MIWAE trained on each of the datasets.
In (´Smieja et al., 2018) a Gaussian Mixture Model, GMM, is used to model the density of the covariates while jointly training a discriminator.
In case of missing values, the discriminator’s ﬁrst input neuron’s activations are set to the average activations as found over a conditional distribution over the missing values in the GMM.
We adapt the ofﬁcial code 2 to this experiment by adjusting the number of hidden layers and number of mixture components while keeping the original training setup as is.
In ﬁgures 10–13 decision surfaces and imputations are shown.
The corresponding quantitative results are in tables 1–4.
[Table] [Table]
[Table] [Table]
[Figure] [Figure]
[Figure] [Figure]
E. C APACITY
The predictive performance of the discriminator at different capacities, in terms of number of hidden units, is shown in ﬁgure 14.
F I MAGE C LASSIFICATION F. T RAINING DETAILS FOR MNIST/ F MNIST
For all MNIST/fMNIST experiments, the discriminative model is a convolutional neural network with four layers and a categorical distribution over classes, see table 5.
The generative model uses a convolutional architecture similar to the DCGAN (Radford et al., 2015) and a latent space of dimension 128.
A Continuous Bernoulli (Loaiza-Ganem & Cunningham, 2019) observation model is used for MNIST and the discretized logistic distribution (Salimans et al., 2017) is used as the observation model for Fashion MNIST.
During training, K = 5 importance samples and a batch size of 128 are used.
[Table]
F. I MPUTATIONS
Figures 15–17 show imputations on the two image datasets for the three different MCAR missing processes.
The ﬁrst column contains data with missing values.
The second column is single imputation by the MIWAE, and following columns are MIWAE multiple imputations using samplingimportance-resampling.
F. N ATURAL IMAGES (SVHN)
In order to assess classiﬁcation of natural images we now turn to the Street View House Numbers dataset (SVHN, Netzer et al., 2011), with different missing mechanisms.
In the “observed squares” missing mechanism 20 × 20 randomly placed squares are observed while in the “missing squares” missing mechanism 16 × 16 randomly placed squares are missing.
In the “random dropout” missing mechanism pixels are missing with probability m = 0 .
In the “observed squares” experiment we apply both a low capacity 4 layer discriminator (1 repetition) and a higher capacity 8 layer discriminator (3 repetitions) using the convolutional gated blocks from Dauphin et al.
(2017), see architectures in tables 7 and 8.
In table 6, test set accuracies are shown and for the 8 layer discriminator, the accuracy on the fully observed test set is also shown.
In the low capacity regime, the supMIWAE outperforms the two single imputation methods and performs on par with MIWAE.
With the higher capacity discriminator supMIWAE has performance similar to 0-imputation and learnable imputation, while the MIWAE imputation is slightly worse.
This displays the same phenomenon as in the 2D classiﬁcation experiments, where powerful learners are able to undo single imputations.
While the discriminative models learnt based on constant imputation can achieve the same performance as the supMIWAE, the kinds of discriminative models that are learnt are different.
In this case, when applied to a fully observed test-set the discriminator learnt by the supMIWAE outperforms the other models.
The high capacity discriminator is also applied to the “missing squares” and “random dropout” missing mechanisms, see ﬁgure 18 for the test-set accuracies (3 repetitions).
In ﬁgure 19 the models trained on data with missing values are applied to a fully observed test-set.
[Figure]
[Table]
G R EGRESSION
All discriminative models are neural networks with one hidden layer and 50 hidden units, where the ﬁnal layer of the neural networks parameterizes a Gaussian distribution.
The datasets are split randomly 20 times with 90% of the data in a training set and 10% in a test set.
A validation set with 10% of the training data is used for early stopping and a batch size of 256 is used.
For the permutation invariant setup, an embedding of dimension U = 20 and ﬁxed length vectors ι PI ( x obs ) of dimension M = 10 are used.
Missing values are introduced completely at random in the training, validation and test sets by removing each feature with probability m
[Table]
where m is the missing rate.
In order to avoid reducing the sample size, due to completely missing covariates, whenever all the covariates of an observation go missing, we set one of them to observed, selected at random
[Table]
[Figure]
[Table]
For the generative models, 2 layered neural networks with 128 hidden units are used for the encoder and the decoder and a latent space of dimension 10 .
A Gaussian observation model is used and
K = 25 importance samples are used during training.
We compare different imputation techniques, modelling approaches and a non-deep competitor.
Zero-imputation, MissForest (Stekhoven & B¨uhlmann, 2012) and MICE (Buuren & GroothuisOudshoorn, 2010) are different imputation techniques applied, before training the discriminative model.
Learnable imputation is using learnable constants as imputation, while permutation invariance is utilizing a learnable embedding.
Finally, histogram-based gradient boosting (Friedman, 2001), as implemented in sklearn (Pedregosa et al., 2011), is a non-deep baseline, using the validation set for early stopping.
Results are seen in ﬁgure 20.
[Figure] [Table]