U NSUPERVISED
C ONDITIONAL G ENERATION USING NOISE ENGINEERED MODE MATCHING GAN
Anonymous authors
Paper under double-blind review
A BSTRACT
Conditional generation refers to the process of sampling from an unknown distribution conditioned on semantics of the data.
This can be achieved by augmenting the generative model with the desired semantic labels, albeit it is not straightforward in an unsupervised setting where the semantic label of every data sample is unknown.
In this paper, we address this issue by proposing a method that can generate samples conditioned on the properties of a latent distribution engineered in accordance with a certain data prior.
In particular, a latent space inversion network is trained in tandem with a generative adversarial network such that the modal properties of the latent space distribution are induced in the data generating distribution.
We demonstrate that our model despite being fully unsupervised, is effective in learning meaningful representations through its mode matching property.
We validate our method on multiple unsupervised tasks such as conditional generation, dataset attribute discovery and inference using three real world image datasets namely MNIST, CIFAR-10 and CelebA and show that the results are comparable to the state-of-the-art methods.
I NTRODUCTION
Unsupervised learning or extraction of semantically relevant information from unlabelled data is a problem of great interest to the machine learning community.
If successfully extracted, such an information can be utilized in a wide range of tasks such as feature learning, supervised and semisupervised classiﬁcation, data augmentation and reconstruction (Hastie et al.
; Hinton et al.
(2006); Hinton & Salakhutdinov (2006); Bengio et al.
; Yann (1987); Radford et al.
Kingma et al.
(2014); Cheung et al.
From a generative model’s perspective, unsupervised learning amounts to dividing the data manifold into semantically interpretable regions and sample new points from each of the regions (Jaakkola & Haussler (1999); Hastie & Tibshirani (1996)).
However, distributions of most of the naturally occurring data such as images and speech lie over complex high-dimensional manifolds and do not have tractable forms.
Thus simple generative models are not found to be successful in modelling such data.
Deep generative neural networks, speciﬁcally, Generative Adversarial Networks (GANs) (Goodfellow et al.
(2016)) have been shown to be highly effective in modelling complex distributions for several real-world tasks (Radford et al. (2015); Isola et al.
; Van Den Oord et al.
Odena et al.
; Denton et al.
The basic idea of a GAN is to learn to implicitly sample from the data generating distribution without explicitly estimating it.
It does so, by adopting an adversarial learning principle where the problem of sampling from an unknown data distribution is formulated as a min-max game between two function approximators called the generator and the discriminator.
A generator has to effectively learn to map an input noise distribution to the data generating distribution.
This is facilitated by the discriminator that is adversarially trained to distinguish between the samples drawn from the true data generating distribution and the samples generated by the generator.
At the point of equilibrium, the distribution of the generated samples converges to the data generating distribution.
GANs are similar to the conventional generative models in that they too are designed to learn a mapping from a latent space (usually noise) to the data space.
However since GANs do not explicitly model the relation between the latent and the data space, they cannot be readily used for unsupervised learning unlike the conventional generative models where the posterior of the latent variable conditioned on the data is tractable.
Thus, if the relation between the latent and the data space can be inferred in a GAN setting, they can be readily used to perform a variety of unsupervised learning tasks.
Owing to this need, several successful recent attempts (although in different contexts) have been made to learn an inverse mapping from the data space to the latent space (Chen et al.
; Donahue et al.
; Dumoulin et al.
; Mescheder et al.
; Larsen et al.
; Makhzani et al.
; Kim et al.
(2017); Springenberg (2015)).
The basic idea is to train a GAN to simultaneously learn to sample from the data distribution and also to approximate the posterior of latent variables conditioned on data.
This technique is also shown to have several desirable side-effects such as stabilization of training and prevention of degenerative solution for the generator (known as mode collapse)
(Che et al. (2016); Srivastava et al.
; Arjovsky et al.
; Salimans et al.
In conventional generative models although, the prior on the latent variable is chosen in accordance with some guess on the structure of the data.
For example, number of clusters in Kmeans is chosen to be ten if there are ten ‘classes’ in the data (Jain et al.
This suggests that engineering the latent (noise) space in accordance with data priors might lead to the possibility of multiple applications.
Motivated by the above observations, in this paper, we propose a GAN model where an inversion mapping from the generated data space to an engineered latent space is learned such that modal properties of the data generating distribution are matched to those of the latent distribution.
The proposed model thus preserves the semantics of the data in the sense that each mode in the generated space corresponds to a certain data semantic.
We demonstrate the use of this method in several unsupervised learning tasks such as conditional generation, inference and attribute discovery through experiments on multiple real-world datasets 1 .
P ROPOSED M ETHOD
Suppose P X denotes the data generating distribution from which the samples are to be drawn and X denote its support.
If X is a disconnected set, which is a union of k non-empty disjoint connected open subsets ( X i , i = { 1 , 2 ...k } ) , each representing a mode, then P X is termed as a multimodal distribution with k modes.
In the following sections of the paper, we describe a procedure to induce multiple modes in the data generating space by engineering the latent space along with inversion.
For the sake of simplicity, we shall assume that P X is a bimodal distribution albeit all the analysis hold equally well for multimodal distributions.
Let z denote a latent variable with a distribution z ∼ P Z and Z be the support of P Z .
Let g : Z → X be a continuous function mapping from z to x which serves as an estimator for P X .
Let X 0 ⊆
X and X 1 ⊆
X represent two modes of P X .
We claim that the following properties have to hold for
P X to be multimodal.
Proposition 1.
Let Z 0 ⊆
Z and Z 1 ⊆
Z respectively be the inverse images of X 0 and X 1 under g , then X 0 ∩ X 1 =
Φ only if Z 0 ∩ Z 1 = Φ , where Φ is an empty set.
Proof:
In Appendix A.
Also since g is a continuous mapping, Z 0 ⊆
Z and Z 1 ⊆
Z also happen to be open subsets.
Thus, from proposition 1, to get a bimodal generated distribution, it is necessary to have a bimodal latent distribution but not sufﬁcient because Proposition 1 is not two way.
However, if there exists a mapping h : X → ˆ Y which maps x , the output of g , to another random variable ˆy in such a way that
P ˆ Y is bimodal, one can expect to apply proposition 1 again on h and claim bi-modality on P X .
The below corollary to proposition 1 is stated to afﬁrm this fact.
Corollary 1.1.
Assume ˆ Y 0 ⊆
ˆ Y and ˆ Y 1 ⊆
ˆ Y are two subsets of ˆ Y such that h : X 0 → ˆ Y 0 and h :
X 1 → ˆ Y 1 then ˆ Y
∩ ˆ Y 1 =
Φ only if X 0 ∩ X 1 = Φ .
Given Z 0 ∩ Z 1 =
Φ , ˆ Y
∩ ˆ Y 1 =
Φ is a sufﬁcient condition for X 0 ∩ X 1 = Φ .
Corollary 1.1 asserts if h maps x to any bimodal distribution, the generated distribution ( P X ) will be bimodal.
However, since both g and h are continuous mappings, conditional generation of samples from P X (conditioned on modes of z ) is possible only if h is in accordance with the modal properties of z .
Thus, we introduce a new random variable y that is an indicator of the modes of z .
Formally, y := 1 z ∈ Z 1 , which implies that P Y ( y = 1)
=
(cid:82) Z 1 P Z d z and P Y ( y = 0)
= (cid:82) Z 0 P Z d z .
With this, we show that if P Y and P ˆ Y are close, then the modal properties of P X will match with those of P Z .
Proposition 2.
Deﬁne a random variable that is an indicator of modes of P X as follows:
ˆ x := 1 x ∈ X 1 ,
P ˆ X (ˆ x = 1)
= (cid:82) X
P X d x and P ˆ X (ˆ x = 0)
= (cid:82) X
P X d x .
The minimization of D KL ( P ˆ Y || P Y ) is equivalent to the minimization of D KL ( P ˆ Y || P ˆ X ) .
Proof: In appendix A.
Corollary 2.1.
Minimizing D KL ( P ˆ Y
|| P
ˆ X ) leads to matching of modal properties of P ˆ X and P Y , that is, (cid:82) X
i P X
d x = P Y ( y = i ) , i ∈ { 0 , 1 } .
Proof:
In Appendix A.
Proposition 2 and Corollary 2.1.
state an important fact - modal properties of the latent space and the data generating space can be matched by choosing h that would minimize D KL ( P ˆ Y || P Y ) .
This also implies that the imbalance (if any) in the modes of the latent distribution is reﬂected in the data generating distribution.
This is extremely useful for datasets containing imbalanced classes, which is often the case in real world applications.
The latent space Z can be constructed following the presumed class imbalance ratio in the real data to conditionally generate imbalanced classes.
Using proposition 1 and 2, one can make the data generating distribution to be bimodal, however, produced modes might be degenerated in the sense that X i ’s reduce to singletons (mode collapse).
To avoid this degenerative case, we propose to decompose h as a composite of two mappings h 1 :
X → ˆ Z and h 2 : ˆ Z → ˆ Y .
Further, minimizing a norm distance between samples of Z and ˆ Z will prevent degenerative modes in P X .
This is because h 2 will enforce unique reconstruction of every sample of z which in turn ensures that a unique sample of x is generated by a unique sample of z .
In the above formulation however, the generated distribution is not constrained to be close to the distribution of the real data.
For that, one has to rely on a technique (such as GANs) that would enforce g ( z ) to be close to the distribution of the real data.
Thus training a GAN according to the proposed formulation, would simultaneously enforce the generated data distribution to be close to the real data distribution and match the modal properties of the latent distribution.
Conversely, if the latent distribution is chosen in accordance with the modal properties of the real data distribution, the generated data space will be clustered conditioned on the modes of the real data.
This is a very desirable property because if the modes of the real data signify semantic separation, then by adopting the proposed procedure one can achieve unsupervised semantically meaningful conditional generation.
In the following sections, we detail the procedure to realize the proposed method in a
GAN framework and validate the claims on a variety of real world tasks.
R EALIZATION USING G ENERATIVE ADVERSARIAL NETWORKS
GAN realizes the mapping g using a deep neural network.
Additionally, to ensure P X is close to the real data distribution a second neural network called the discriminator denoted by d is adversarially trained to differentiate between a true data sample and generated data sample.
At the optimum, the generated distribution, P X approaches the distribution of the real data, P X r .
We utilize a GAN to realize g and the inversion network
h = h 1 ◦ h 2 is also modelled using deep neural networks.
All neural networks, g , d and h are simultaneously trained to optimize the following objective function:
[Equation]
The variables and notations in Equation 1 are consistent with the ones in Section 2.
The proposed method can be seen as establishing a chain of continuous functions of random variables - y
→ z → x → ˆ z → ˆ y and imposing a similar modal structures on all of them.
The model being completely unsupervised offers several advantages - (a) Once trained, sampling z from a particular mode conﬁnes x to a unique mode leading to conditional generation, (b) when training by imposing a ﬁxed prior on z , unseen attributes in the data can be discovered, (c) post training, the h network can be used independently to cluster the data, (d) if z is well-reconstructed by h 1 , this architecture can be used for inference (Note that this is not guaranteed since the model convergence can be deﬁned only via convergence of h 2 even if z is not reconstructed back by h 1 ), (e) Mode-collapse when generated
P X becomes independent of the latent random variable z is avoided in our formulation since g is explicitly forced to produce a multi-modal distribution by the inversion network h .
R ELATED WORK Data space auto-encoders
In these class of methods, the idea is to learn a mapping from the data to the latent space and back to the data space.
Methods such as vanilla autoencoders (AE) (Hinton & Salakhutdinov (2006), variational autoencoders (VAE) Kingma & Welling (2013), adverarial autoencoders (AAE) Makhzani et al.
, mode regularized GANs (Che et al.
, VAE/GAN (Larsen et al. (2015)) fall into this category.
While these methods offer several advantages, there are two major issues with this idea -(a) without regularization, a naive AE model ends up learning an identity mapping.
Further, a vanilla autoencoder cannot be used as a generative model because latent distribution is not tractable and hence cannot be sampled from, (b) although VAE, AAE, VAE/GAN circumvent this problem by using either variational inference or adversarial training, these models cannot ﬁnd disentangled mappings and thus cannot be used for unsupervised conditional generation tasks.
Latent space auto-encoders
In an alternative setting, these class of methods essentially learn mappings from the latent space to the data space and backwards, autoencoding the latent/noise space.
These methods have an advantage over the data autoencoders in that it is relatively easy to reconstruct the noise vectors as one has control over the distribution from which they are chosen.
Methods such as Adversarially learned inference (ALI) (Dumoulin et al.
(2016)), Bidirectional GANs (BiGAN) (Donahue et al.
VEEGAN Srivastava et al.
and structured GANs (Deng et al.
(2017)) are examples of this approach.
In ALI and BiGAN, an encoder is learned that would map the data space to the latent space and additionally the discriminator is taken to classify between the tuples of encoded data-real data and latent noise-generated data.
However in these methods, the latent space encoding is learned independently of the generated data, in the sense that the real-data is inverted back but not the generated data.
This is shown to lead to poor quality reconstruction and possibility of mode collapse in the generated data space Srivastava et al.
VEEGAN on the other hand maps both the real and generated data space to a ﬁxed distribution in order to prevent the problem of mode collapse.
While these methods are more advantageous over data-space autoencoders for the task at hand, they still do not learn disentangled relation between the data and the latent space as they invert the data to a semantically-meaningless ﬁxed noise distribution.
Regularized information maximizers
Methods such as InfoGAN (Chen et al.
and CaTGAN (Springenberg (2015)) propose to train the GAN along with a regularization term maximizing the mutual information between the generated data and parts of the latent space, following the idea of regularized information maximization (Krause et al.
These methods are shown to learn disentangled mappings between the latent and the data space in a completely unsupervised manner albeit they might lead to mode collapse because parts of latent space are not reconstructed.
Further since there are no explicit function mapping for noise reconstruction, the learned semantic disentanglement may not be completely faithful (Kim & Mnih (2018); Ghosh et al.).
Also inference in the data space is not possible because a major part of the latent space is ignored while information maximization.
4. O UR CONTRIBUTIONS
Engineering latent/noise space - Almost all the aforementioned methods use unimodal distribution such as Gaussian or Uniform distribution for the latent space.
As we have shown in Proposition 1, multiple modes are produced in the data generating space when the latent space has multiple modes.
Further, even if reconstructed perfectly, a
unimodal noise cannot be used for conditional generation since the semantic relation between the latent and the data space is unknown.
In this work we propose to engineer the noise space according to the guess one has on the real data distribution.
The speciﬁcs for the choice of noise space will depend upon the task and the guess one has on the data prior and will be described in the experiment section.
Two-stage latent reconstruction - The latent-space reconstruction network is made a composite of two functions ( h 1 and h 2 ) one for norm-based reconstruction ( g ( h 1 ( x r ) used in inference) and one to only reproduce the modal properties ( g ( z ) used for conditional generation).
Among them, the KL-loss contributes to mode matching and the norm-loss to the variety in the generated data.
Devoid of approximate/variational inference - In many formulations such as InfoGAN and VEEGAN, the loss involves an entropy term that requires sampling from an unknown (posterior of the latent variable given the real data) distribution which cannot be computed.
These methods rely on minimizing a lower bound on the actual cost via variational inference.
However, in our case both the loss terms are exact and hence there is no need for constructing bounds.
Based on the above observations, we call our approach Noise Engineered Mode-matching GAN (NEMGAN).
During implementation, a multimodal noise is created by adding uniform noise with a multinomial random variable (akin to y ) which is selected in accordance to the assumed data prior.
The resulting distribution will be a convolution product of the two distributions resulting in a multimodal distribution with non-overlapping modes.
E XPERIMENTS
We conduct three sets of experiments on three real world datasets namely MNIST (LeCun et al.
, CelebA (Liu et al. (2015)) and CIFAR-10 (Krizhevsky & Hinton (2009)), to evaluate our claims.
(a) Mode matching - Design noise space in accordance with the semantic imbalance ratio in the data and evaluate conditional generation, (b) Attribute discovery - Impose a ﬁxed but arbitrary prior on noise and see if the generated modes are semantically meaningful, (c) Semantic representation learning - No particular prior is imposed on noise except for the number of modes and evaluate if meaningful representations are learned and (d) Data inference - validate the data reconstruction capabilities of NEMGAN.
Note that all experiments are fully unsupervised.
5. M ODE MATCHING
We take two semantically ‘distinct’ classes (digits 0 and 4) for MNIST based on their t-SNE plots (Maaten & Hinton (2008)), albeit this paradigm is applicable to any combination of digits.
Three imbalance ratios of 50:50, 25:75, and 02:98 are considered, signifying no imbalance, slight imbalance and heavy imbalance, respectively in digit 0 and 4.
Similar class imbalance is induced in P Y and hence in the latent space P Z , by construction.
Figure 1 depicts the images generated by NEMGAN corresponding to two noise modes.
It can be seen that even with an imbalance ratio of 02:98, the network is able to conditionally generate semantically separated images.
Further to illustrate the mode matching, we plot the class-wise histograms of the distances between the individual samples and their corresponding means in Figure 2, for both the noise and the generated data space.
It can be seen that there is evidence for mode matching between latent and generated spaces.
As another level of sanity check, we evaluate the classiﬁcation accuracy of the reconstruction network on the real data without imbalance h ( x r ) and observed accuracy of about 99% for ﬁrst two and 90% for the case with a class imbalance of 02:98 in the training data.
Next to evaluate the semantic separation, we consider images corresponding to digits 3 and 5, which have a signiﬁcant overlap in the t-SNE with an imbalance ratio of 30:70.
The left pane of Figure 3 shows the conditional generation for the considered classes (3 and 5) which shows no overlap between the classes in spite of their closeness in the data manifold.
In fact, the inversion network h ( x r ) achieves a classiﬁcation accuracy of 98.
% on real test samples of the digits 3 and 5 in these settings.
We conduct the mode matching experiments with complex color data sets - CIFAR-10 and CelebA.
CIFAR in particular is known to have very high intra-class variability.
First, we consider two relatively distinct classes namely airplane and frogs, with an imbalance ratio of 30:70.
Once again, the modes are matching leading to conditional generation results shown in right pane of Figure 3 with an inversion network classiﬁcation accuracy of 81.05% on unseen real data.
Similarly, the facial images generated conditioned on presence and absence of glasses, when the latent space is induced with an imbalance of 07:93 are shown in ﬁrst two panes of Figure 4.
Furthermore, for an imbalance of 05:95 the network learns to latch on to another facial attribute namely presence or absence of rosy
[Figure] [Figure]
cheeks as observed in last two panes of Figure 4.
Another interesting observation in all these experiments is that there is considerable amount of visual variety in the images from each class suggesting reduction of mode collapse even within a single mode with 2% occurrence.
These experiments thus demonstrate the ability of NEMGAN to perform an unsupervised generation conditioned on semantically meaningful classes through mode matching.
To end the section on mode matching we evaluate our method on standard mode-counting tasks.
Speciﬁcally, as mentioned in Srivastava et al.
(2017) we used stacked MNIST dataset which is obtained by stacking three random MNIST digits along color channels of an RGB image to obtain color images.
This dataset can have 1000 modes corresponding to every possible combination of
[Figure]
digit triplets.
We used experimental settings identical to Srivastava et al.
Table 1 lists the results of our method and three other methods on discovering the modes in the stacked MNIST.
For our method, latent space is a discrete uniform distribution with 1000 modes.
It can be seen that
NEMGAN is able to discover 993 modes out of 1000 modes which is quite signiﬁcant.
Appendix
D shows generated images and also depicts the outcome of NEMGAN on a eight component GMM arranged over a circle in which case NEMGAN discovers all the modes.
[Table]
5. A TTRIBUTE DISCOVERY
In a few real-life scenarios, the class imbalance ratio is unknown apriori.
In such cases, an unsupervised technique should discover semantically plausible regions in the data space.
To evaluate NEMGANs ability to perform such a task, we perform experiments where sample from P Y are drawn with an assumed class ratio rather than a known ratio.
Two experiments are performed on CelebA, ﬁrst with the assumption of 2 classes having a ratio of 70:30 and the second with the assumption of 3 classes having a ratio of 10:30:60.
In the ﬁrst experiment, the network discovers visibility of teeth as an attribute to the faces whereas in the second it learns to differentiate between the facial pose angles.
Conditional generation from both the experiments are shown in ﬁgure 5 and 6, respectively.
Note that these attributes are not labelled in the dataset but are discovered by NEMGAN.
5. S EMANTIC REPRESENTATION LEARNING
In these experiments, we use a P Z that has k uniform modes and train NEMGAN on MNIST and
CIFAR-10.
In the ﬁrst experiment, k was set to 10 for both the datasets based on the fact that there are 10 uniformly distributed semantically distinct classes in both of them.
Figure 7 depicts the sample from generated images using the MNIST dataset - it can be seen that every generated mode correspond to a digit type.
Further a classiﬁcation accuracy of 96 . 09% was obtained on the real data from h ( x r ) .
To quantify the separability, we compute the class-wise KL divergence between the real and the generated digits (for each class) using a nearest neighbor density estimation technique (Duda et al.
(2012)), shown in Figure 7.
It can be seen that each mode (class) in the generated data has a unique digit with which the KL is least, signifying a digit-wise semantic separation.
To quantify the goodness of the generated data, we compute ”Frchet Inception Distance” (FID) which is a measure proposed to captures the closeness of the generated images to real data Heusel et al.
Tablelists the FID obtained after 5 runs for NEMGAN and two comparable models, namely, VEEGAN and InfoGAN.
It can been seen that NEMGAN achieves the least FID that is close to the real MNIST digits signifying the goodness of the images generated.
[Table]
Next we repeat the experiment with k = 10 on CIFAR-10.
Given the huge intra and inter class variety in the CIFAR it is difﬁcult for a fully unsupervised method to generate ten semantically separated modes.
However, as seen in the left most pane of Figure 8, NEMGAN is surprisingly able to generate ten modes with every mode dominated by one CIFAR class.
Further we conducted the
CIFAR experiment with k = 5 (5 modes) taking ﬁve semantically ‘distinct’ classes from CIFAR (horses, airplanes, dogs, frogs and automobiles) and the results are shown in the middle pane of
Figure 8.
It can be again seen that class-wise separation is qualitatively more in this case as expected since the classes are ‘more’ distinct which is also corroborated from the class-wise KL plot in the right-most pane of Figure 8.
To the best of our knowledge, ours is the ﬁrst study to report unsupervised class-conditional generation on CIFAR images generated and real data.
5. A BLATION STUDIES
Our model involves two reconstruction stages for the latent space ( h 1 and h 2 ).
In this section, to study the effect of individual components of the model, we perform the following ablation studies - (a) training a NEMGAN without the h 1 network, (b) training a NEMGAN without the h 2 network, (c) training a conventional GAN with noise engineering.
Experiment (a) and (b) are conducted on the MNIST dataset and the output images are showed in Fig 9.
It can be seen that the absence of the KL-loss ( h 1 ) results in the mixing of classes within each and the absence of norm-loss ( h 2 ) results in lack of variety within each mode.
For example, 1’s with serifs are not generated in absence of h 2 .
Results for the experiment (c) is depicted in the Appendix E which suggests that a conventional GAN with latent space engineering cannot separate out the modes.
These experiments suggests that the inclusion of the norm-based reconstruction term encourages the model to the
5. I NFERENCE
In the ﬁnal set of experiments, though not the main objective of the paper, we qualitatively evaluate
NEMGAN on an inference task where the real images are reconstructed using the outputs of the norm ( L 1 , used in this work) loss layer.
Figure 10 depicts the inference for arbitrarily chosendigits from MNIST and CIFAR samples from the two class (Frog Vs.
Airplane) experiment (more images with qualitative comparison is given in the Appendix).
We have observed that the semantic category is almost always preserved.
Given that the network was trained with a skewed class ratio of 70-30, it is signiﬁcant that a class-wise faithful reconstruction is obtained.
[Figure]
C ONCLUSION
We introduced a method for unsupervised conditional generation using generative adversarial networks through engineered latent space inversion.
We demonstrated through multiple experiments, that certain desirable properties could be induced in the generated data by creating a latent noise space that is engineered in accordance to the need.
Being fully unsupervised, NEMGAN can be used in multiple applications spanning from conditional generation to data space clustering.
It also has several desirable side effects like reduction of the infamous mode collapse.
Even though NEMGAN is fully unsupervised, we believe its performance can be improved by augmenting it with some supervision.
A. Proofs for Propositions
Proposition 1: Let Z 0 ⊆
Z and Z 1 ⊆
Z respectively be the inverse images of X 0 and X 1 under g , then X 0 ∩ X 1 =
Φ only if Z 0 ∩ Z 1 = Φ , where Φ is an empty set.
Proof: Given X 0 ∩ X 1 = Φ , assume, Z
∩ Z 1 (cid:54) =
Φ = ⇒ ∃ z i ∈
Z 0 ∩ Z 1 .
Given z i ∈ Z 0 , let g ( z i ) =
x i 0 ∈ X 0 and similarly, given z i ∈
Z 1 , g ( z i ) =
x i 1 ∈ X 1 .
Since g is a continuous function, x i
= x i 1 = x i = ⇒ x i ∈
X
∩ X
1 contradicting the fact that X 0 ∩ X 1 = Φ , hence Z 0 ∩ Z 1 = Φ .
Proposition 2: Deﬁne a random variable that is an indicator of modes of P X as follows: ˆ x := 1 x ∈ X 1 ,
P ˆ X (ˆ x = 1)
= (cid:82) X
P X d x and P ˆ X (ˆ x = 0)
= (cid:82) X
P X d x .
The minimization of D KL ( P ˆ Y || P Y ) is equivalent to the minimization of D KL ( P ˆ Y || P ˆ X ) .
Proof: As we know
[Equation]
Since (cid:82) Z i P Z d z = (cid:82) X i P X d x , equation 3 can be written as
[Equation]
x
[Equation]
Since (cid:82) X i P X
d
x
= P
ˆ X (ˆ x = i ) , by deﬁnition, equation 6 can be written as
[Equation]
Corollary 2.1: Minimizing D KL ( P ˆ Y
|| P
ˆ X ) leads to matching of modal properties of P ˆ X and P Y , that is, (cid:82) X
i P X
d x = P Y ( y = i ) , i ∈ { 0 , 1 } .
Proof: The optimum value of D KL ( P ˆ Y ||
P
ˆ X ) is achieved when P ˆ X (ˆ x = i ) =
P Y ( y = i ) .
Further from equation 9, P Y ( y = i )
=
P ˆ Y (ˆ y = i ) .
Hence, P ˆ X (ˆ x = i ) =
P Y ( y = i ) , i ∈ { 0 , 1 } .
B.1. Qualitative Comparison with existing architectures
[Figure]
B.2. Inference comparison
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
C. Mode Separation
In this work, semantics of the data refer to the modes in data distribution.
These semantics represent different attributes of the samples and are separated out by the proposed method.
For a better understanding, experiments are conducted with samples of only a single digit type from the MNIST dataset.
Samples of digit 7 and 4 are considered for this purpose.
NEMGAN is trained with a discrete uniform latent space with 10 modes and the generated images are shown in Fig.
Each row in Fig.
16 corresponds on one latent space mode and shows different attributes of the considered digits.
For example, the ﬁfth row in left pane contains generated images of digit 7 with slits.
Similarly in right pane, the third row contains images of digit 4 with a closed notch.
Note that, even with images of a single digit, no mode collapse is observed with NEMGAN.
We trained NEMGAN for mode counting experiment on stacked MNIST dataset.
It is able to generate 993 modes.
Some of the generated images are shown in Fig.
Similar performance is observed in 8 component GMM experiment, as shown in Fig.
E. Conventional GAN with latent space engineering
Proposition 1 and 2 show that if there exists a mapping from the generated data space back to the latent space, the generated space is deemed to have as many modes as in the latent space.
Multimodal latent space is necessary to generate date with multiple modes, however, it is not sufﬁcient.
To understand this, the mode matching experiment performed with images of digits 3 and 5 in section 5.1 is repeated with conventional GAN.
Images of digit 3 and 5 with an imbalance ratio of 30: are used to train the conventional GAN with a bimodal latent space having a modal mass of 30:70.
Oservations of the experiment are shown in Fig.
The conventional GAN is not able to separate the modes and most of the samples drawn are from the dominant mode Fig.
19(b).
On the other hand, NEMGAN separates out the modes accurately.
[Figure] [Figure]
F. Conditional generation of CIFAR images
G. Network architectures used in the work
[Table] [Table]
[Table] [Table]
[Table] [Table]