L EARNING FROM I NCOMPLETE D ATA WITH G ENERATIVE A DVERSARIAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative adversarial networks (GANs) have been shown to provide an effective way to model complex distributions and have obtained impressive results on various challenging tasks.
However, typical GANs require fully-observed data during training.
In this paper, we present a modular approach to learning GANs from incomplete observations that can be combined with different generator and discriminator networks and is amenable for use with complex, high-dimensional inputs.
The proposed framework learns a complete data generator along with a mask generator that models the missingness.
We further demonstrate how to impute missing data by equipping our framework with an adversarially trained imputer.
We evaluate the proposed framework using a series of experiments with several types of missing completely at random missing data processes
I NTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al., 2014) provide a powerful modeling framework for learning complex high-dimensional distributions.
Unlike likelihood-based methods,
GANs are referred to as implicit probabilistic models (Mohamed & Lakshminarayanan, 2016).
They represent a probability distribution through a generator that learns to directly produce samples from the desired distribution.
The generator is trained adversarially by optimizing a minimax objective together with a discriminator.
In practice, GANs have been shown to be very successful in a range of applications including generating photorealistic images (Karras et al., 2018).
Other than generating samples, many downstream tasks require a good generative model, such as image inpainting (Pathak et al., 2016; Yeh et al., 2017).
Training GANs normally requires access to a large collection of fully-observed data.
However, it is not always possible to obtain a large amount of fully-observed data.
Missing data is well-known to be prevalent in many real-world application domains where different data cases might have different missing patterns.
This arbitrary missingness can poses a signiﬁcant challenge when learning certain kinds of models, including essentially all discriminative models.
Following Little & Rubin (2014), the generative process for incompletely observed data can be described below where x ∈ R n is a complete data vector and m ∈ { 0 , 1 } n is a binary mask 1 that determines which entries in x to reveal:
[Equation]
Let x obs denotes the observed elements of x and x mis the missing elements according to the mask m .
In addition, let θ denote the unknown parameters of the data distribution, and φ denote the unknown parameters for the mask distribution, which are usually assumed to be independent of θ .
In the standard maximum likelihood setting, the unknown parameters are estimated by maximizing the following marginal likelihood, integrating over the unknown missing data values:
[Equation]
Little & Rubin (2014) characterize the missing data mechanism p φ ( m | x obs , x mis ) in terms of independence relations between complete data and the masks:
• Missing completely at random (MCAR): p φ ( m | x ) =
p φ ( m ) , • Missing at random (MAR): p φ ( m | x ) =
p φ ( m | x obs ) , • Not missing at random (NMAR): m depends on x mis in addition to x obs .
Most work on incomplete data assumes MCAR or MAR because under these assumptions p ( x obs , m ) can be factorized into p θ ( x obs ) p φ ( m | x obs ) .
With such decoupling, the missing data mechanism can be ignored when learning the data generating model while yielding correct estimates for θ .
When p θ ( x ) does not admit efﬁcient marginalization over x mis , estimation of θ is usually performed by maximizing a variational lower bound, as shown below, using the EM algorithm or a more general approach (Little & Rubin, 2014; Ghahramani & Jordan, 1994):
[Equation]
The primary contribution of this paper is the development of a a framework called MisGAN for learning the distribution from high-dimensional data in the presence of incomplete observations.
Our framework augments a standard GAN with an auxiliary generator and discriminator to learn the mask distribution for modeling the missingness.
The framework learns to generate both complete data examples (as in a standard GAN), and the masks that indicate which entries to be observed.
The generated masks are used to “mask” the generated complete data example by ﬁlling elements indicated as missing with a constant value.
The learning criterion is based on matching the distribution of masked generated data examples to an empirical distribution of the masked data examples, obtained by similarly ﬁlling in the missing values in the incompletely observed real data examples with the same constant value.
Further, the learning criterion requires that the distribution of the masks match the empirical distribution of the masks associated with the incomplete data.
Our framework builds on the ideas of AmbientGAN (Bora et al., 2018).
AmbientGAN modiﬁes the discriminator of a GAN to distinguish corrupted real samples from corrupted generated samples under a range of corruption processes (or measurement processes).
For images, examples of the measurement processes include random dropout, blur, block-patch, and so on.
Missing data can be seen as a special type of corruption, except that we have access to the missing pattern in addition to the corrupted measurements.
Moreover, AmbientGAN assumes the measurement process is known or parameterized only by a few parameters, which is not the case in general missing data problems.
We provide empirical evidence that the proposed framework is able to effectively learn complex, high-dimensional data distributions from highly incomplete data when the structure of the GAN generator provides sufﬁciently strong constraints over the data generation process.
We further show how the architecture can be used to generate high-quality imputations.
M IS GAN: GAN FOR MISSING DATA
In the missing data problem, we know exactly which entries in each data examples are missing.
Therefore, we can represent an incomplete data case as a pair of a partially-observed data vector x ∈ R n and a corresponding mask m ∈ { 0 , 1 } n that indicates which entries in x are observed: x d is observed if m d = 1 otherwise x d is missing and might contain an arbitrary value that we should ignore.
With this representation, an incomplete dataset is denoted D
=
( x i , m i ) } i =1 ,...,N , assumed i.i.d.
We choose this representation instead of x obs because it leads to a cleaner description of MisGAN.
It also suggests how MisGAN can be implemented efﬁciently in practice as both x and m are ﬁxed-length vectors.
We begin by deﬁning a masking operator f τ that ﬁlls in missing entries with a constant value τ , where
¯ m denotes the complement of m and (cid:12) denotes element-wise multiplication:
[Equation]
Two key ideas underlie the MisGAN framework.
First, since the masks in the incomplete dataset are fully observed, we can estimate their distribution using any generative model applicable to fully-observed data.
Second, we use two generators, one for generating complete data vectors and the other for generating masks.
We combine the output of the two generators with the GAN training framework using f τ to match the distribution of the masked incompletely observed real data (masked with the same f τ ) and the masked complete generated data.
Speciﬁcally, we use two generator/discriminator pairs ( G m , D m ) and ( G x , D x ) for the mask and the data respectively.
In this paper, we focus on the case of missing completely at random, where the two generators are independent of each other and have their own noise distributions p z and p ε .
We deﬁne the following two loss functions, one for the masks and the other for the data:
[Equation]
The losses above follow the Wasserstein GAN formulation (Arjovsky et al., 2017), although the proposed framework is compatible with many GAN variations (Goodfellow et al., 2014; Berthelot et al., 2017; Gulrajani et al., 2017).
We optimize the generators and the discriminators according to the following objectives:
[Equation]
where F x , F m are deﬁned such that D x , D m are both 1-Lipschitz as in Wasserstein GANs (Arjovsky et al., 2017).
Practically, we follow the common practice that alternates between a few steps of optimizing the discriminators and one step of optimizing the generators (Goodfellow et al., 2014;
Arjovsky et al., 2017; Gulrajani et al., 2017).
The coefﬁcient α is introduced when optimizing the mask generator G m to aim at minimizing both L m and L x .
Although in theory we could choose α = 0 to train G m and D m using only masks without the data x , we ﬁnd that choosing a small value such as α = 0 .
2 gives better performance.
This encourages the generated masks not only match the distribution of the real masks, but under which the masked generated complete samples also attempt to match the masked real data.
The overall structure of MisGAN is illustrated in Figure 8 in
Appendix C.
Note that the data discriminator D x takes as input the masked samples as if the data are fullyobserved.
This allows us to use any existing architecture designed for complete data to construct the data discriminator.
There is no need to develop customized neural network modules for dealing with missing data.
For example, D x can be a standard convolutional network for image applications.
Note that the masks are binary-valued.
Since discrete data generating processes have zero gradient almost everywhere, to carry out gradient-based training for GANs, we relax the output of the mask generator G m from { 0 , 1 } n to [0 , 1] n .
We employ a sigmoid activation σ λ ( x ) = 1 / (1 + exp( − x/λ )) with a low temperature 0 < λ < 1 to encourage saturation and makes the output closer to zero or one.
Finally, we note that as result of these modeling choices, the discriminator D x in MisGAN is unaware of which entries are missing in the masked input samples, and does not even need to know which value τ is used for masking.
In the next section, we present a theoretical analysis providing support for the idea that this type of masking process does not necessarily make it more difﬁcult to recover the complete data distribution.
The experiments provide compelling empirical evidence of the effectiveness of the proposed framework.
T HEORETICAL RESULTS
In Section 2 we mentioned that the discriminator D x in MisGAN takes as input the masked samples using (2) without knowing what value τ is used and which entries in the input vector are missing.
In this section, we are going to discuss the following two important questions: i)
How should we choose the ﬁlled-in value τ ? ii) Does ignoring the information about the location of missing values complicate the problem in terms of the chance of recovering the data distribution?
We address these questions in a simpliﬁed scenario where each feature in the data vector takes values from a ﬁnite set P .
For n -dimensional data, let M = { 0 , 1 } n be the set of masks and I =
P n be the set of all possible data vectors.
Also let D M and D I be the set of probability distributions on M and
I respectively, so all the elements in them are non-negative and sum to one.
We ﬁrst discuss the case where the ﬁlled-in value τ ∈ P as it is the more convenient case in practice.
Given τ ∈ P and q ∈ D M , we can construct a left transition matrix T q ,τ ∈ R I×I deﬁned below where the ( t , s ) -th entry speciﬁes the transition probability from a data vector s ∈ I to an outcome t ∈ I masked by f τ , which involves all possible masks under which s is converted into t by ﬁlling in with τ :
[Equation]
Let p ∗ x ∈ D
I be the unknown true data distribution we want to estimate.
In the presence of missing data speciﬁed by q , the masked samples then follow the distribution p y
=
T q ,τ p ∗ x .
Without making use of any prior knowledge about the underlying application, MisGAN with α = 0 in (6) can be viewed as a method for solving the linear system p y
=
T q ,τ p x , where p x ∈ D I is the unknown data distribution to solve for.
Here we consider that p y and T q ,τ are given, as those can be estimated separately from a collection of fully-observed masks and masked samples.
Note that a transition matrix preserves the sum of the vectors it is applied to since 1 (cid:62) T q ,τ = 1 (cid:62) .
For p x to be a valid distribution vector, we only need the non-negativity constraint because any solution p x automatically sums to one.
That is, estimating the data generating process in the presence of missing data based on the masking scheme as in MisGAN is equivalent to solving the linear problem
[Equation]
We ﬁrst state a key property of the transition matrix T q ,τ that leads to the answer to our questions.
Theorem 1.
Given q ∈ D M , the transition matrices T q ,τ have the same null space for all τ ∈ P .
The proof is deferred to Appendix A. Theorem 1 implies that if the solution to the constrained linear system (7) is not unique for a given τ 0 ∈ P , that is, there exists some non-negative p x (cid:54) =
p ∗ x such that
T q ,τ 0 p x = T q ,τ 0 p ∗ x , then we must have T q ,τ p x = T q ,τ p ∗ x for all τ ∈ P .
In other words, whether the true data distribution is uniquely recoverable is independent of the choice of the ﬁlled-in value τ
and therefore we can arbitrarily choose any value for τ ∈ P .
Here we only discuss the case when the probability of observing all features q ( 1 ) is zero; otherwise the linear system is uniquely solvable as the transition matrix T q ,τ 0 has full rank.
With the nonnegativity constraint, it is possible that the solution for the linear system (7) is unique when the true data distribution p ∗ x is sparse.
In particular, if there exists two indices s 1 , s 2 ∈
I such that v ( s 1 ) >and v ( s 2 ) < 0 for all v ∈ Null ( T q ,τ ) \ { 0 } and also p ∗ x ( s 1 ) =
p ∗ x ( s 2 ) = 0 , then the solution to (7) is unique, where v ( s ) denotes the (scalar) entry in v indexed by s .
Roughly speaking, when the missing rate is high, that is, if the masks in M that have many zeros are more probable, the null space of T q ,τ will be larger and therefore it is more likely that the non-negative solution is not unique.
The sparsity of the data distribution is a reasonable assumption in many situations.
For example, natural images are typically considered to lie on a low dimensional manifold, which means most of the instances in I should have almost zero probability.
Bruckstein et al.
(2008) proposed a sufﬁcient condition on the sparsity of the non-negative solutions to a general underdetermined linear system that guarantees unique optimality.
In the case of τ ∈ P , an entry τ in some masked sample t ∈ I might have the original entry being τ or it might be masked into τ due to the missingness in that entry.
One might wonder if this prevents an algorithm from recovering the true distribution if that is possible.
In other words, if we take the location of the missing values into account, would that make the missing data problem less ill-posed?
Theorem 1 suggests that those concerns are not the case with following:
Corollary 1.
If the linear system T q ,τ p x = T q ,τ p ∗ x does not have a unique non-negative solution, then for this missing data problem we cannot uniquely recover the true data distribution even if we take the location of the missing values into account.
Corollary 1 is justiﬁed in Appendix B, in which we discuss the case for τ / ∈ P .
Note that the analysis above characterizes how difﬁcult the missing data problem is, which is independent of the choice of the algorithm that solves it.
In practice, it is common to incorporate prior knowledge into the model to regularize the problem when it is ill-posed.
For example, for modeling natural images, convolutional networks are commonly used to exploit the local structure of the data.
In addition, decoder-based deep generative models such as GANs implicitly enforce some (soft) sparsity constraints due to the use of low dimensional latent code for the generator, which also helps to regularize the problem.
M ISSING DATA IMPUTATION
Missing data imputation is an important technique when dealing with incomplete data.
In this section, we show how to impute missing data according to p ( x mis | x obs ) by equipping MisGAN with an imputer G i accompanied by a corresponding discriminator D i .
The imputer is a function of the incomplete example ( x , m ) and a random vector ω drawn from a noise distribution p ω .
It outputs the completed sample with the observed part in x kept intact.
To train the imputer-equipped MisGAN, we deﬁne the loss for the imputer in addition to (3) and (4):
[Equation]
We jointly learn the data generating process and the imputer according to the following objectives:
[Equation]
where we use β = 0 .
1 in the experiments when optimizing G x for encouraging the generated complete data to match the distribution of the imputed real data in addition to having the masked generated data to match the masked real data.
The overall structure of MisGAN is illustrated in
Figure 9 in Appendix C.
We can also train a stand-alone imputer using only (8) with a pre-trained data generator G x as the architecture shown in Figure 9 without the shaded part.
Moreover, it is also possible to train the imputer targeting a different missing distribution p m with a pre-trained data generator G x alone without access to the original (incomplete) training data:
[Equation]
We construct the imputer G i ( x , m , ω ) as follows:
[Equation]
where f IMP generates the imputed result with the same dimensionality as its input, x (cid:12) m + ω (cid:12) ¯ m , which could be implemented by a deep neural network.
The masking outside of f IMP ensures that the observed part of x stays the same in the output of the imputer G i .
The similar masking on the input of f IMP , x (cid:12) m + ω (cid:12) ¯ m , ensures that the amount of noise injected to f IMP is complementary to the size of the observed features.
This is intuitive in the sense that when a data case is almost fully-observed, we expect less variety in p ( x mis | x obs ) and vice versa.
Note that the noise ω needs to have the same dimensionality as x .
E XPERIMENTS
In this section, we ﬁrst assess various properties of MisGAN on the MNIST dataset: we demonstrate qualitatively how MisGAN behaves under different missing patterns and different architectures.
We then conduct an ablation study to justify the construction of MisGAN.
At the end, we compare
MisGAN with various baseline methods on the missing data imputation task over three datasets under a series of missingness settings.
Data We evaluate MisGAN on three datasets: MNIST, CIFAR-10 and CelebA. MNIST is a dataset of handwritten digits images of size 28 × 28 (LeCun et al., 1998).
We use the provided 60, training examples for the experiments.
CIFAR-10 is a dataset of 32 × 32 color images from 10 classes (Krizhevsky, 2009).
Similarly, we use 50,000 training examples for the experiments.
CelebA is a large-scale face attributes dataset (Liu et al., 2015) that contains 202,599 face images, where we use the provided aligned and cropped images and resize them to 64 × 64.
For all three datasets, the range of pixel values of each image is rescaled to [0 , 1] .
Missing data distributions We consider three types of missing data distribution: i) Square observation : all pixels are missing except for a square occurring at a random location on the image.
ii) Dropout : each pixel is independently missing according to a Bernoulli distribution.
iii)
Variable– size rectangular observation : all pixels are missing except for a rectangular observed region.
The width and height of the rectangle are independently drawn from 25% to 75% of the image length uniformly at random, which results in a 75% missing rate on average.
As opposed to the previous two missing distributions that have a constant missing rate, in this missing distribution each example may have different number of missing pixels.
In the extreme case, the missing rate can go up to 93.75%.
Evaluation metric We use the Fréchet Inception Distance (FID) (Heusel et al., 2017) to evaluate the quality of the learned generative model.
Instead of the Inception network trained on ImageNet (Salimans et al., 2016), we use a basic LeNet implementation 2 trained on the complete MNIST training set.
We take the 50-dimensional output from the second-to-last fully-connected layer as the features and compute the FID for MNIST.
For CIFAR-10 and CelebA, we follow the procedure described in Heusel et al.
(2017) to compute the FID using the pretrained Inception-v3 model.
When evaluating the generative model using the FID, we use the same number of samples generated by the model as the size of the training set.
5. E MPIRICAL STUDY OF M IS GAN ON MNIST
In this section, we study various properties of MisGAN perform on the MNIST dataset.
Architectures We consider two kinds of architecture for MisGAN: convolutional networks and fully connected networks.
We follow the DCGAN architecture (Radford et al., 2015) for (de)convolutional generators and discriminators to exploit the local structures of images.
We call this model ConvMisGAN.
To demonstrate the performance of MisGAN in the absence of the implicit structural regularization provided by the use of a convolutional network, we construct another MisGAN with only fullyconnected layers for both the generators and the discriminators, which we call FC-MisGAN.
In the experiments, both Conv-MisGAN and FC-MisGAN are trained using the improved procedure for the Wasserstein GAN with gradient penalty (Gulrajani et al., 2017).
Throughout we use τ =for the masking operator the temperature λ = 0 . 66 for the mask activation σ λ ( x ) as described in
Section 2.
Baseline We compare MisGAN to a strong baseline model for learning from large-scale incomplete data: generative convolutional arithmetic circuit (ConvAC) models (Sharir et al., 2016).
ConvAC is an expressive mixture model similar to sum-product networks (Poon & Domingos, 2011) with a compositional structure similar to deep convolutional networks.
Moreover, ConvAC admits tractable marginalization due to the product form of the base distributions for the mixtures, which makes it readily capable for learning with missing data.
Results Figures 1 and 2 show the generated data samples as well as the learned mask samples produced by Conv-MisGAN and FC-MisGAN under the square observation and independent dropout missing mechanisms.
From these results, we can see that Conv-MisGANs produces visually better samples than FC-MisGANs.
On the other hand, under the same missing rate, independent dropout leads to worse samples than square observations.
Samples generated by ConvAC are shown in
Figure 16 in Appendix G.
We quantitatively evaluate Conv-MisGAN, FC-MisGAN and ConvAC under two missing patterns with missing rates from 10% to 90% with a step of 10%.
Figure 3 shows that MisGAN in general outperforms ConvAC as ConvAC tends to generate samples with aliasing artifacts as shown in
Figure 16.
It also shows that in the square observation case, Conv-MisGAN and FC-MisGAN have similar performance in terms of FIDs.
However, under independent dropout, the performance of
FC-MisGAN degrades signiﬁcantly as the missing rate increases compared to Conv-MisGAN.
This is because independent dropout with high missing rate makes the problem more challenging as it induces less overlapping co-occurrence among pixels, which degrades the signal for understanding the overall structure.
This is illustrated in Figure 4 where the observed pattern comes from one of four equally probable 14 × 14 square quadrants with no overlap.
Clearly this missing data problem is ill-posed and we could never uniquely determine the correlation between pixels across different quadrants.
The samples generated by the FC-MisGAN produce obvious discontinuity across the
[Figure] [Figure]
boundary of the quadrants as it does not rely on any prior knowledge about how pixels are correlated.
On the contrary, the discontinuity artifact is less severe with Conv-MisGAN as the convolutional layers encourage local smoothness.
This shows the importance of incorporating prior knowledge into the model when the problem is highly ill-posed.
Ablation study We point out that the mask discriminator in MisGAN is important for learning the correct distribution robustly.
Figure 5 shows two common failure scenarios that frequently happen with an AmbientGAN
[Figure]
which is essentially equivalent to a MisGAN without the mask discriminator.
Figure 5 (left) shows a case where AmbientGAN produces perfectly consistent masked outputs
[Figure]
but the learned mask distribution is completely wrong.
This is a coincidence as both the mask and pixel values are in the same range [0 , 1] in our experiments.
Since we use f τ =0 ( x , m )
= x (cid:12) m , it makes the role of x and m interchangeable.
To avoid this situation, we rescale the range of pixel values from [0 , 1] to [ − 1 , 1]
[Figure]
but AmbientGAN still fails often.
Figure 5 (right) shows another frequently occurring failure case of AmbientGAN.
In contrast, MisGAN avoids learning such degenerate solutions due to explicitly modeling the mask distribution.
[Figure] [Figure]
[Figure]
Missing data imputation We construct the imputer network f IMP using a a three-layer fullyconnected network with 500 hidden units in all the middle layers.
Figure 6 (left) shows the imputation results on different examples applying novel masks randomly drawn according to the same distribution.
Figure 6 (right) shows the imputation results where each row corresponds to the same incomplete input.
It demonstrates that the imputer can produce a variety of different imputed results due to the random noise input to the imputer.
We also note that if we modify (11) to train the imputer together with the data generator from scratch without the mask generator/discriminator, it fails most of the time for a similar reason to why AmbientGAN fails.
The learning problem is highly ill-posed without the constraints on the mask distribution.
5. Q UANTITATIVE EVALUATION
In this section, we quantitatively evaluate the performance of MisGAN on three datasets: MNIST,
CIFAR-10, and CelebA.
We ﬁrst note that we found ConvAC, which we compare MisGAN with in Section 5.1, fails to learn the distribution of CIFAR-10 and CelebA as it overﬁts severely when it applies to color images as pointed out in Sharir et al.
(2016) even with suggested regularization; see Appendix G for more details.
Therefore, we focus on evaluating MisGAN on the missing data imputation task as it is widely studied and many baseline methods are available.
Baselines We compare MisGAN imputer to a range of baseline methods including the basic zero/mean imputation, matrix factorization, and the recently proposed Generative Adversarial Imputation Network (GAIN)
(Yoon et al., 2018).
GAIN is an imputation model that employs an imputer
[Figure]
network to complete the missing data.
It is trained adversarially with a discriminator that determines which entries in the completed data were actually observed and which were imputed.
It has shown to outperforms many state-of-the-art imputation methods
[Figure]
Evaluation of imputation We impute all of the incomplete examples in the training set and use the
FID between the imputed data and the original fully-observed data as the evaluation metric.
Architecture We use convolutional generators and discriminators for MisGAN for all experiments in this section.
For MNIST, we use the same fully-connected imputer network as described in the previous section; for CIFAR-10 and CelebA, we use a ﬁve-layer U-Net architecture (Ronneberger et al., 2015) for the imputer network f IMP in MisGAN.
Results We compare all the methods under two missing patterns, square observation and independent dropout, with missing rates from 10% to 90%.
Figure 7 shows that MisGAN consistently outperforms other methods in all of the cases especially under high missing rate.
In our experiments, we found
GAIN training is quite unstable for the block observation missingness.
There is a sweet spot for the number of training epochs for training GAIN.
If trained longer, the imputation behavior will gradually become similar to constant imputation; see Appendix H for more details.
On the other hand, training
MisGAN is more stable than GAIN across all scenarios in the experiments.
The imputation results of
MisGAN and GAIN are shown in Appendix E, F, and H.
D ISCUSSION AND FUTURE WORK
This work presents and evaluates a highly ﬂexible framework for learning standard GAN data generators in the presence of missing data.
Although we only focus on the MCAR case in this work,
MisGAN can be easily extended to cases where the output of the data generator is provided to the mask generator.
These modiﬁcations can capture both MAR and NMAR mechanisms.
The question of learnability requires further investigation and the analysis in Section 3 no longer holds due to the dependence between the transition matrix and the data distribution.
However, we have tried this modiﬁed architecture in our experiments and it showed similar results as to the original MisGAN.
This suggests that the extra dependencies do not adversely affect learnability.
We leave the formal evaluation of this modiﬁed framework for future work.
A P ROOF OF T HEOREM
Let P be the ﬁnite set of feature values.
For the n -dimensional case, let M = { 0 , 1 } n be the set of masks and I =
P n be the set of all possible feature vectors.
Also let D M be the set of probability distributions on M , which implies m (cid:23) 0 and (cid:80) v ∈I m ( v )
= 1 for all m ∈ M , where m ( v ) denotes the entry of m indexed by v .
Given τ ∈ P and q ∈ D M , deﬁne the transformation
T q ,τ : R I → R I x
(cid:55)→ y
=
T q ,τ x by y ( v ) = T q ,τ x ( v ) =
(cid:88) m ∈M (cid:88)
u ∈I q ( m ) x ( u ) 1 { u (cid:12) m
+ τ ¯ m = v } , for all v ∈ I (12) where (cid:12) is the entry-wise multiplication and 1 {·} is the indicator function.
Given m ∈ M , deﬁne an equivalent relation ∼ m on I by v ∼ m u iff v (cid:12) m = u (cid:12) m , and denote by [ v ] m the equivalence class containing v .
Given q ∈ D M , let S q ⊂ M be the support of q , that is,
S q = { m ∈ M : q ( m ) > 0 } .
Given τ ∈ P and v ∈ I , let M τ, v denote the set of masks consistent with v in the sense that q ( m ) > 0 and v (cid:12) ¯ m = τ ¯ m , that is,
M τ, v = { m ∈ S q : v (cid:12) ¯ m =
τ ¯ m } .
Proposition 1.
For any q ∈
D M and x ∈
R I , the collection of marginals { x ([ v ] m ) : v ∈
I , m ∈ S q } determines T q ,τ x for all τ ∈ P where x ([ v ] m ) :
= (cid:80) u ∈ [ v ] m x ( u ) .
Proof.
This is clear from the following equation
[Equation]
which can be obtained from (12) as follows,
[Equation]
Proposition 2.
For any τ ∈ P , q ∈ D M and x ∈ R
I , the vector T q ,τ x determines the collection of marginals { x ([ v ] m ) : v ∈ I , m ∈ S q } .
Proof.
Fix τ ∈ P , q ∈ D M and x ∈ R
I .
Since v (cid:12) m + τ ¯ m ∈
[ v ] m , it sufﬁces to show that we can solve for x ([ v ] m ) in terms of T q ,τ x for m ∈ M τ, v (cid:54) = ∅ .
We use induction on the size of M τ, v .
First consider the base case |M τ, v | = 1 .
Consider v 0 ∈
I with M τ, v 0 = { m 0 } .
By (13),
T q ,τ x ( v 0 )
=
q ( m 0 ) x
([ v 0 ] m 0 ) .
Hence x ([ v 0 ] m 0 )
=
T q ,τ x ( v 0 ) /
q ( m 0 ) , which proves the base case.
Now assume we can solve for x ([ v ] m ) in terms of T q ,τ x for m ∈ S q and v ∈ I with |M τ, v | ≤ k .
Consider v 0 ∈ I with |M τ, v 0 | = k + 1 ; if no such v 0 exists, the conclusion holds trivially.
Let
M τ, v 0 = { m 0 , m 1 , . . .
, m k } .
We need to show that T q ,τ x determines x ([ v 0 ] m (cid:96) ) for (cid:96) = 0 , 1 , . . .
, k .
By (13) again,
[Equation]
[Equation]
Let m =
(cid:86) k(cid:96)
=
m (cid:96) , which may or may not belong to S q .
Note that
[Equation]
and hence x
[Equation]
Plugging (15) into (14) yields
[Equation]
Note that M τ, v ⊂ M τ, v 0 \ { m (cid:96) } for v ∈ [ v 0 ] m ∨ ¯ m (cid:96) \ { v 0 } , so |M τ, v | ≤ k .
By the induction hypothesis, x ([ v ] m (cid:96) ) is determined by T q ,τ x .
It follows from (16) and (15) that x ([ v 0 ] m ) and x ([ v 0 ] m (cid:96) ) are also determined by T q ,τ x .
This completes the induction step.
Theorem 1 is a direct consequence of Proposition 1 and Proposition 2 as the collection of marginals { x ([ v ] m ) : v ∈ I , m ∈ S q } is independent of τ .
Therefore, if x 1 , x 2 ∈ R I satisfy T q ,τ 0 x 1 =
T q ,τ 0 x 2 for some τ 0 ∈ P , then T q ,τ x 1 = T q ,τ x 2 for all τ ∈ P .
Theorem 1 is a special case when x 1 = 0 .
Moreover, Proposition 2 also shows that MisGAN overall learns the distribution p ( x obs , m ) , as x ([ v ] m ) is equivalent to p ( x obs | m ) and T q ,τ x is essentially the distribution of f τ ( x , m ) under the optimally learned missingness q = p ( m ) .
This is also true when τ / ∈ P according to Appendix B.
B J USTIFICATION OF C OROLLARY
Corollary 1 can be explained by augmenting the set of feature values by P (cid:48)
=
P ∪ { φ } with a novel symbol φ / ∈ P .
If we choose τ = φ for the masking operator, whenever we spot a φ in a masked sample, we know that it corresponds to a missing entry.
We can also construct the corresponding transition matrix T (cid:48) q ,φ ∈ R
I (cid:48) ×I (cid:48) where I (cid:48)
=
( P (cid:48) )
n given the mask distribution q ∈ D M before.
In this setting, the generative model for missing data is equivalent to solving the linear system
T q ,φ p (cid:48) x
= T q ,φ p ∗ x (cid:48)
so that p (cid:48) x ∈ R
I
(cid:48) is non-negative and p (cid:48)
x ( s ) =
for all s ∈ I (cid:48) \ I , where the true distribution p ∗ x (cid:48) is given by p ∗ x (cid:48) ( s ) =
p ∗ x ( s ) for all s ∈ I and zeros elsewhere.
Theorem 1 implies that if the solution to original problem (7) is not unique, the non-negative solution to the augmented linear system with the extra constraint on I (cid:48) \ I with τ = φ is not unique either.
C A RCHITECTURE OF M IS GAN AND IMPUTER
Figure 8 and Figure 9 illustrate the architecture of MisGAN and the MisGAN imputer.
D A RCHITECTURAL DETAILS AND HYPERPARAMETERS
All of the generators and discriminators in Conv-MisGAN follow the architecture used by the
DCGAN model (Radford et al., 2015) with 128-dimensional latent code.
As for FC-MisGAN, the architecture of the generators is
[Equation]
with ReLUs in between.
The discriminators are of the structure
[Equation]
also with ReLUs in between.
For the imputer network for MisGAN trained on CIFAR-10 and CelebA, we follow the U-Net implementation of the CycleGAN and pix2pix work 3 .
In the experiments, we use 5-layer U-Nets for both CIFAR-10 and CelebA.
For training Wasserstein GAN with gradient penalty, We use all the default hyperparameters reported in Gulrajani et al.
For all the datasets, MisGAN is trained for 300 epochs.
We train MisGAN imputer for 1000 epochs for MNIST and CIFAR-10 as the networks are smaller and 600 epochs for
CelebA.
For ConvAC, we use the same architecture described in Sharir et al.
We train ConvAC for 1000 epochs using Adam optimizer with learning rate 10 − 4 .
[Figure]
E M IS GAN ON CIFAR-
Figure 10, 11 and 12 show the results of MisGAN trained on CIFAR-10 for the two extreme missing rates, namely 90% and 80%, as well as the case of 10% that is close to full observation.
F M IS GAN ON C ELEB A
Figure 13, 14 and 15 show the results of MisGAN trained on CelebA for the two extreme missing rates, namely 90% and 80%, as well as the case of 10% that is close to full observation.
G R ESULTS OF C ONV AC
Figure 16 shows the samples generated by ConvAC trained with the square observation missing pattern.
However, we ﬁnd that when training ConvAC on CIFAR-10, the generated samples have severe aliasing artifacts and fail to capture the details in the training images as shown in Figure 17.
On the other hand, ConvAC is hard to scale up to data like CelebA. For CelebA, if we use 256 channels for the ﬁrst layer, it requires 50,331,648 parameters for the input image of size 64 × 64.
H M ISSING DATA IMPUTATION WITH GAIN
Figure 18 shows the imputation results of GAIN on different epochs during training with the 20 × 20 square observation missingnss.
We found that this is a common phenomenon for the square observation missing pattern.
To obtain better results for GAIN, we analysis the FIDs during the course of training and use the model that achieves the best FID to favorably compare with MisGAN for the square observation case.
For CIFAR-10, we use the results from the 500th epoch; for CelebA, [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
we use the results from the 50th epoch.
Otherwise, we train GAIN for 1000 epochs for CIFARand 300 epochs for CelebA.
Our implementation is adapted from the code released by the authors of
GAIN.
Figure 19 shows the imputation results of GAIN for both CIFAR-10 and CelebA.
I R ELATIONSHIP BETWEEN IMPUTATION AND IMAGE INPAINTING
Different from most of the image inpainting work that aims at completing an incomplete image and producing a single visually convincing result, the goal of missing data imputation is to model the conditional distribution p ( x mis | x obs ) .
In addition, most of the state-of-the-art image inpainting work are required to train with complete image datasets (Pathak et al., 2016; Yeh et al., 2017; Liu et al., 2018).
We also note that although we demonstrate a simple construction of the imputer G i in the MisGAN imputation framework as described in Section 4, it is compatible with many specialized architectures that suit the application of interest.
When it comes to image inpainting, we could use the architectures of other image inpainting work (Pathak et al., 2016;
Yeh et al., 2017; Liu et al., 2018)
as the imputer network within our framework.
J E VALUATION OF IMPUTATION USING ROOT MEAN SQUARE ERROR
Root mean square error (RMSE) is a commonly used metric for evaluating the performance of missing data imputation, which computes the RMSE of the imputed missing values against the ground truth.
However, in a complex system, the conditional distribution p ( x mis | x obs ) is likely to be highly multimodal.
It’s not guaranteed that the ground truth of the missing values in the incomplete dataset created under the missing completely at random (MCAR) assumption correspond to the [Figure] [Figure]
[Figure] mode of p ( x mis | x obs ) . A good imputation model might produce samples from p ( x mis | x obs ) associated with a higher density than the ground truth (or from other modes that are similarly probable).
In this case [Figure] it will lead to a large error in terms of metrics like RMSE as multiple modes might be far away from each other in a complex distribution.
Therefore, we instead compute the FID between the distribution of the completed data and the distribution of the originally fully-observed data as our evaluation metric.
This provides a practical way to assess how close a model imputes according to p ( x mis | x obs ) by comparing two groups of samples collectively.
As a concrete example [Figure] Figure 20 compares the two evaluation metrics on MNIST, our distributionbased FID and the ground truth-based RMSE.
It shows that the rankings on most of the missing rates are not consistent across the two metrics.
In particular, under 90% missing rate, MisGAN is worse than GAIN and matrix factorization in terms of RMSE, but signiﬁcantly better in terms of FID.
Figure 21 plots the imputation results of the three methods mentioned above.
We can clearly see that
MisGAN produces the best completion even though its RMSE is much higher than the other two.
It’s not surprising as the mean of p ( x mis | x obs ) minimizes the squared error in expectation, even if the mean might have low density.
This probably explains why the blurry completion results produced by matrix factorization achieve the lowest RMSE.