Original: T OWARDS E FFECTIVE GAN S FOR D ATA D ISTRIBUTIONS WITH D IVERSE M ODES
Revised: T OWARDS E FFECTIVE GAN S FOR D ATA D ISTRIBUTIONS WITH D IVERSE M ODES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generative Adversarial Networks (GANs), when trained on large datasets with diverse modes, are known to produce conﬂated images which do not distinctly belong to any of the modes.
Revised: Generative Adversarial Networks (GANs), when trained on large datasets with diverse modes, are known to produce conﬂated images which do not distinctly belong to any of the modes.

Original: We hypothesize that this problem occurs due to the interaction between two facts: (1) For datasets with large variety, it is likely that each mode lies ona separate manifold.
Revised: We hypothesize that this problem occurs due to the interaction between two facts: (1) For datasets with large variety, it is likely that the modes lie on separate manifolds.

Original: The generator (G) is(often) formulated as a continuous function, and the input noise is derived from a connected set, due to which G’s output must be a connected set.
Revised: The generator (G) is formulated as a continuous function, and the input noise is derived from a connected set, due to which G’s output is a connected set.

Original: If G covers all modes, then there must be some portion of G’s output which connects them.
Revised: If G covers all modes, then there must be some portion of G’s output which connects them.

Original: This corresponds to undesirable, conﬂated images.
Revised: This corresponds to undesirable, conﬂated images.

Original: We develop theoretical arguments to support these intuitions.
Revised: We develop theoretical arguments to support these intuitions.

Original: We propose a novel method to break the second assumption via learnable discontinuities in the latent noise space.
Revised: We propose a novel method to break the second assumption via learnable discontinuities in the latent noise space.

Original: Equivalently, it can be viewed as training several generators, thus creating discontinuities in the G function.
Revised: Equivalently, it can be viewed as training several generators, thus creating discontinuities in the G function.

Original: We also augment the GAN formulation with a classiﬁer C that predicts which noise partition/generator produced the output images,thus encouraging diversity between each partition/generator.
Revised: We also augment the GAN formulation with a classiﬁer C that predicts which noise partition/generator produced the output images, encouraging diversity between each partition/generator.

Original: We experiment on MNIST, celebA,  and a difﬁcult dataset with clearly distinct modes, and show that the noise partitions correspond to different modes of the data distribution, and produce images of superior quality.
Revised: We experiment on MNIST, celebA, STL-10, and a difﬁcult dataset with clearly distinct modes, and show that the noise partitions correspond to different modes of the data distribution, and produce images of superior quality.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative Adversarial Networks (Goodfellow et al., 2014) are powerful generative models that have enjoyed signiﬁcant attention from the research community in the past few years.
Revised: Generative Adversarial Networks (Goodfellow et al., 2014) are powerful generative models that have enjoyed signiﬁcant attention from the research community in the past few years.

Original: Despite several successes, the original formulation for GANs is widely acknowledged to be notoriously difﬁcult to train due to instability issues.
Revised: Despite several successes, the original formulation for GANs is widely acknowledged to be notoriously difﬁcult to train due to instability issues.

Original: In particular, GANs face the mode collapse problem, where the generator resorts to generating a handful of samples which are assigned high probability by the discriminator.
Revised: In particular, GANs face the mode collapse problem, where the generator resorts to generating a handful of samples which are assigned high probability by the discriminator.

Original: Several methods have been introduced to ﬁx the mode collapse problem.
Revised: Several methods have been introduced to ﬁx the mode collapse problem.

Original: (Che et al.
Revised: (Che et al.

Original: (2016), Arjovsky et al.
Revised: (2016), Arjovsky et al.

Original: (2017), Gulrajani et al.
Revised: (2017), Gulrajani et al.

Original: (2017), Mao et al.
Revised: (2017), Mao et al.

Original: (2016), Sun et al.
Revised: (2016), Sun et al.

Original: Despite improvements, state-of-art GANs still fail to generate meaningful samples on diverse and complex datasets such as ImageNet (Deng et al., 2009).
Revised: Despite improvements, state-of-art GANs still fail to generate meaningful samples on diverse and complex datasets such as ImageNet (Deng et al., 2009).

Original: GANs trained on such datasets produce conﬂated images which do not distinctly belong to any of the modes present in the dataset.
Revised: GANs trained on such datasets produce conﬂated images which do not distinctly belong to any of the modes present in the dataset.

Original: We hypothesize that this problem occurs due to the continuous nature of the generator function, along with the connectedness of the latent noise space, due to which the output set of the generator is also connected.
Revised: We hypothesize that this problem occurs due to the continuous nature of the generator function, along with the connectedness of the latent noise space, due to which the output set of the generator is also connected.

Original: This poses a problem when dealing with complex real life datasets with varied modes.
Revised: This poses a problem when dealing with complex real life datasets with varied modes.

Original: Strong empirical and theoretical evidence suggests that real life images lie on lowdimensional manifolds (Narayanan & Mitter, 2010).
Revised: Strong empirical and theoretical evidence suggests that real life images lie on lowdimensional manifolds (Narayanan & Mitter, 2010).

Original: It is highly probable that distinct modes (say bedroom images and human face images) lie on disjoint manifolds.
Revised: It is highly probable that distinct modes (say bedroom images and human face images) lie on disjoint manifolds.

Original: If we assume that the generator does not suffer from the mode dropping problem, it must cover all these manifolds in its output.
Revised: If we assume that the generator does not suffer from the mode dropping problem, it must cover all these manifolds in its output.

Original: However, the output set being connected, must contain parts which do not belong to any of the manifolds, but simply join them.
Revised: However, the output set being connected, must contain parts which do not belong to any of the manifolds, but simply join them.

Original: We refer to such parts of the output as tunnels, since they connect otherwise disjoint manifolds.
Revised: We refer to such parts of the output as tunnels, since they connect otherwise disjoint manifolds.

Original: Tunnels do not resemble any of the images in the dataset, and are not similar to any of the modes.
Revised: Tunnels do not resemble any of the images in the dataset, and are not similar to any of the modes.

Original: They correspond to the conﬂated images produced by the generator, and are undesirable.
Revised: They correspond to the conﬂated images produced by the generator, and are undesirable.

Original: By this reasoning, we suggest that GANs with continuous generators and connected latent noise sets must suffer either from a certain degree of mode dropping or from producing conﬂated, garbled outputs when trained on complex and varied datasets like ImageNet.
Revised: By this reasoning, we suggest that GANs with continuous generators and connected latent noise sets must suffer either from a certain degree of mode dropping or from producing conﬂated, garbled outputs when trained on complex and varied datasets like ImageNet.

Original: We develop methods that allow GANs to cover disjoint manifolds without the use of tunnels, while not compromising on mode coverage.
Revised: We develop methods that allow GANs to cover disjoint manifolds without the use of tunnels, while not compromising on mode coverage.

Original: Our approach is to create learnable discontinuities in the latent noise space.
Revised: Our approach is to create learnable discontinuities in the latent noise space.

Original: This is done by learning N different linear mappings (partitions) in the input layer of the generator.
Revised: This is done by learning N different linear mappings (partitions) in the input layer of the generator.

Original: A noise vector (sampled from the standard normal distribution), gets mapped to N different vectors by the input layer, and the rest of the processing remains the same as in standard generators.
Revised: A noise vector (sampled from the standard normal distribution), gets mapped to N different vectors by the input layer, and the rest of the processing remains the same as in standard generators.

Original: The output set of each mapping is a connected set, but the union of the N output sets could potentially be disconnected.
Revised: The output set of each mapping is a connected set, but the union of the N output sets could potentially be disconnected.

Original: Thus, we break the connectedness assumption leading to the existence of tunnels.
Revised: Thus, we break the connectedness assumption leading to the existence of tunnels.

Original: To facilitate learning distinct modes by each partition, we introduce a classiﬁer that predicts which partition created a given input.
Revised: To facilitate learning distinct modes by each partition, we introduce a classiﬁer that predicts which partition created a given input.

Original: We modify the loss functions to adjust for this change.
Revised: We modify the loss functions to adjust for this change.

Original: We experiment on standard datasets: MNIST (LeCun et al., 2010), celebA (Liu et al., 2015),  and a tough artiﬁcial dataset with very distinct modes - an equal mixture of LSUN (Yu et al., 2015) bedrooms and celebA, to verify the efﬁcacy of our method.
Revised: We experiment on standard datasets: MNIST (LeCun et al., 2010), celebA (Liu et al., 2015), STL- (a subset of ImageNet) (Coates et al., 2011), and a tough artiﬁcial dataset with very distinct modes - an equal mixture of LSUN (Yu et al., 2015) bedrooms and celebA, to verify the efﬁcacy of our method.

Original: We compare our results with one of the best performing GAN variant (Gulrajani et al., 2017), and show an improvement in quality.
Revised: We compare our results with one of the best performing GAN variant (Gulrajani et al., 2017), and show an improvement in quality.

Original: The major contributions of the paper are summarized below:
Revised: The major contributions of the paper are summarized below:

Original: We identify a key problem with training GANs on large & diverse datasets, and provide intuition to explain its cause 2.
Revised: We identify a key problem with training GANs on large & diverse datasets, and provide intuition to explain its cause 2.

Original: We develop theoretical analyses to support and introduce rigor in the intuitions provided 3.
Revised: We develop theoretical analyses to support and introduce rigor in the intuitions provided 3.

Original: Motivated by these analyses, we introduce a novel GAN setup to alleviate the problem 4.
Revised: Motivated by these analyses, we introduce a novel GAN setup to alleviate the problem 4.

Original: We experiment on a variety of standard datasets and report improvements over state-of-art formulations
Revised: We experiment on a variety of standard datasets and report improvements over state-of-art formulations

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2014) formulated GAN as a minimax game between two neural networks: generator G θ and discriminator D φ .
Revised: (2014) formulated GAN as a minimax game between two neural networks: generator G θ and discriminator D φ .

Original: G θ takes a random noise vector z as input and generates sample
Revised: G θ takes a random noise vector z as input and generates sample

Original: G θ ( z ) , while D φ identiﬁes whether input sample is real or generated by the generator G θ .
Revised: G θ ( z ) , while D φ identiﬁes whether input sample is real or generated by the generator G θ .

Original: Both G θ and D φ play a two-player minimax game with value function V ( G, D ) :
Revised: Both G θ and D φ play a two-player minimax game with value function V ( G, D ) :

Original: [Equation]
Revised: [Equation]

Original: where P r ( x ) is the real data distribution, and P ( z ) is arbitrary noise distribution (typically uniform or normal distribution).
Revised: where P r ( x ) is the real data distribution, and P ( z ) is arbitrary noise distribution (typically uniform or normal distribution).

Original: In practice, training GANs using above formulation is highly unstable and requires careful balance of generator and discriminator updates.
Revised: In practice, training GANs using above formulation is highly unstable and requires careful balance of generator and discriminator updates.

Original: Radford et al.
Revised: Radford et al.

Original: (2015) proposed a class of CNNs called DCGANs (Deep Convolutional GANs) with certain architectural speciﬁcations, and demonstrated better image quality than non-convolutional vanilla GAN architecture.
Revised: (2015) proposed a class of CNNs called DCGANs (Deep Convolutional GANs) with certain architectural speciﬁcations, and demonstrated better image quality than non-convolutional vanilla GAN architecture.

Original: Denton et al.
Revised: Denton et al.

Original: (2015) used Laplacian pyramid framework for the generator, where a separate generative convnet model is trained using GAN approach at each level of pyramid, to generate images in coarse-to-ﬁne fashion.
Revised: (2015) used Laplacian pyramid framework for the generator, where a separate generative convnet model is trained using GAN approach at each level of pyramid, to generate images in coarse-to-ﬁne fashion.

Original: Despite better architectures, GANs suffered from problems like unstable training, vanishing gradients of generator, mode collapse.
Revised: Despite better architectures, GANs suffered from problems like unstable training, vanishing gradients of generator, mode collapse.

Original: Salimans et al.
Revised: Salimans et al.

Original: (2016) proposed several heuristics such as feature matching, minibatch discrimination, historical averaging, label smoothing, primarily to stabilize
Revised: (2016) proposed several heuristics such as feature matching, minibatch discrimination, historical averaging, label smoothing, primarily to stabilize

Original: GAN training.
Revised: GAN training.

Original: Che et al.
Revised: Che et al.

Original: (2016) observed that GAN training can push probability mass in wrong direction, hence are prone to missing modes of data.
Revised: (2016) observed that GAN training can push probability mass in wrong direction, hence are prone to missing modes of data.

Original: They proposed regularization techniques to stabilize GAN training and alleviate mode missing problem by fair distribution of probability mass across modes of the real data distribution.
Revised: They proposed regularization techniques to stabilize GAN training and alleviate mode missing problem by fair distribution of probability mass across modes of the real data distribution.

Original: Arjovsky & Bottou (2017) provided theoretical analysis of training dynamics of GANs, and problems including instability and saturation.
Revised: Arjovsky & Bottou (2017) provided theoretical analysis of training dynamics of GANs, and problems including instability and saturation.

Original: They revealed fundamental problems with original GAN formulation and provided directions towards solving them.
Revised: They revealed fundamental problems with original GAN formulation and provided directions towards solving them.

Original: Several papers proposed alternative objective function of generator and discriminator.
Revised: Several papers proposed alternative objective function of generator and discriminator.

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: (2017), Gulrajani et al.
Revised: (2017), Gulrajani et al.

Original: (2017) proposed new loss function which approximately minimizes
Revised: (2017) proposed new loss function which approximately minimizes

Original: Wasserstein distance between real and generated data distribution instead of Jensen Shannon Divergence.
Revised: Wasserstein distance between real and generated data distribution instead of Jensen Shannon Divergence.

Original: They claim their formulation does not require careful balance between generator and discriminator updates, thus lead to stable training without saturating the gradients.
Revised: They claim their formulation does not require careful balance between generator and discriminator updates, thus lead to stable training without saturating the gradients.

Original: They observed no evidence of mode collapse in their experiments.
Revised: They observed no evidence of mode collapse in their experiments.

Original: Mao et al.
Revised: Mao et al.

Original: (2016) used squared-loss instead of log-loss in original formulation, which provides generator with better non-vanishing gradients.
Revised: (2016) used squared-loss instead of log-loss in original formulation, which provides generator with better non-vanishing gradients.

Original: Zhao et al.
Revised: Zhao et al.

Original: (2016) view discriminator as an energy function making it possible to use additional loss functions other than logistic output binary classiﬁer, which was found to stabilize GAN training.
Revised: (2016) view discriminator as an energy function making it possible to use additional loss functions other than logistic output binary classiﬁer, which was found to stabilize GAN training.

Original: Sun et al.
Revised: Sun et al.

Original: (2017) propose to train discriminator based on linear separability between hidden representation of real and generated samples and train generator based on decision hyperplanes between hidden representations computed using Linear Discriminant Analysis.
Revised: (2017) propose to train discriminator based on linear separability between hidden representation of real and generated samples and train generator based on decision hyperplanes between hidden representations computed using Linear Discriminant Analysis.

Original: For labelled datasets, Mirza & Osindero (2014), Odena et al.
Revised: For labelled datasets, Mirza & Osindero (2014), Odena et al.

Original: (2016) employed label conditioning in both generator and discriminator to generate discriminable and diverse samples across classes.
Revised: (2016) employed label conditioning in both generator and discriminator to generate discriminable and diverse samples across classes.

Original: While this helps produce better samples for complex datasets, it requires the presence of labelled data.
Revised: While this helps produce better samples for complex datasets, it requires the presence of labelled data.

Original: In this paper we propose methods to improve performance of GANs on complex datasets without making use of labels.
Revised: In this paper we propose methods to improve performance of GANs on complex datasets without making use of labels.

Original:  P ROBLEMS WITH L EARNING D IVERSE D ATASETS
Revised:  P ROBLEMS WITH L EARNING D IVERSE D ATASETS

Original: In this section, we further develop the ideas from the introduction.
Revised: In this section, we further develop the ideas from the introduction.

Original: We also provide theoretical analyses to lend support to these ideas.
Revised: We also provide theoretical analyses to lend support to these ideas.

Original: 3. T OPOLOGY OF D IVERSE D ISTRIBUTIONS
Revised: 3. T OPOLOGY OF D IVERSE D ISTRIBUTIONS

Original: Theoretical analyses and empirical studies suggest that probability distributions of real images (denoted by P r ) have supports that lie on low dimensional manifolds (Arjovsky & Bottou (2017),
Revised: Theoretical analyses and empirical studies suggest that probability distributions of real images (denoted by P r ) have supports that lie on low dimensional manifolds (Arjovsky & Bottou (2017),

Original: Narayanan & Mitter (2010)).
Revised: Narayanan & Mitter (2010)).

Original: We choose to represent the support set S of distribution P r by the set of its connected components, i.e. the set of maximal connected subsets of S .
Revised: We choose to represent the support set S of distribution P r by the set of its connected components, i.e. the set of maximal connected subsets of S .

Original: These components are disjoint and their union is S .
Revised: These components are disjoint and their union is S .

Original: In other words, they form a partition of S .
Revised: In other words, they form a partition of S .

Original: As suggested earlier, each component is a lowdimensional manifold in high-dimensional space.
Revised: As suggested earlier, each component is a lowdimensional manifold in high-dimensional space.

Original: Throughout this paper, we use the terms manifold of S and connected component of S interchangeably, unless mentioned otherwise.
Revised: Throughout this paper, we use the terms manifold of S and connected component of S interchangeably, unless mentioned otherwise.

Original: Consider highly distinct images m 1 , m 2 of a complex and varied dataset.
Revised: Consider highly distinct images m 1 , m 2 of a complex and varied dataset.

Original: These images cannot be path-connected, since that would imply the existence of a smooth sequence of images in S starting from m 1 , and slowly transitioning to m 2 .
Revised: These images cannot be path-connected, since that would imply the existence of a smooth sequence of images in S starting from m 1 , and slowly transitioning to m 2 .

Original: Such a sequence clearly does not exist for very distinct m 1 , m 2 , e.g. in a dataset consisting of face and bedroom images, if m 1 is a face, and m 2 is a bedroom, it is not possible for a sequence of realistic facial and bedroom images to smoothly transition from m 1 to m 2 .
Revised: Such a sequence clearly does not exist for very distinct m 1 , m 2 , e.g. in a dataset consisting of face and bedroom images, if m 1 is a face, and m 2 is a bedroom, it is not possible for a sequence of realistic facial and bedroom images to smoothly transition from m 1 to m 2 .

Original: Since path-connectedness and connectedness are equivalent properties in open subsets of R n , the manifolds on which m 1 , m 2 lie, must be disconnected, and hence must be separate components of S .
Revised: Since path-connectedness and connectedness are equivalent properties in open subsets of R n , the manifolds on which m 1 , m 2 lie, must be disconnected, and hence must be separate components of S .

Original: We summarize the above discussion with the following results:
Revised: We summarize the above discussion with the following results:

Original: Result 1.
Revised: Result 1.

Original: Sufﬁciently distinct images from the support set of real-life image probability distributions must lie on disconnected manifolds.
Revised: Sufﬁciently distinct images from the support set of real-life image probability distributions must lie on disconnected manifolds.

Original: As a corollary, we have:
Revised: As a corollary, we have:

Original: Result 2.
Revised: Result 2.

Original: The support set of a real-life image probability distribution must have at least as many connected components as the size of the maximal set of (sufﬁciently, pairwise) distinct modes.
Revised: The support set of a real-life image probability distribution must have at least as many connected components as the size of the maximal set of (sufﬁciently, pairwise) distinct modes.

Original: 3. T OPOLOGY OF G ENERATOR ’ S O UTPUT S ET
Revised: 3. T OPOLOGY OF G ENERATOR ’ S O UTPUT S ET

Original: The generator is often formulated as a continuous function of the input latent noise vector.
Revised: The generator is often formulated as a continuous function of the input latent noise vector.

Original: Moreover, the probability distribution from which the noise vector is drawn is usually a simple distribution like
Revised: Moreover, the probability distribution from which the noise vector is drawn is usually a simple distribution like

Original: Gaussian or uniform, which have connected support sets.
Revised: Gaussian or uniform, which have connected support sets.

Original: Finally, we observe that a continuous function, applied to a connected set results in an output set which is connected.
Revised: Finally, we observe that a continuous function, applied to a connected set results in an output set which is connected.

Original: Thus we have:
Revised: Thus we have:

Original: Result 3.
Revised: Result 3.

Original: The output set produced by a continuous generator acting on a latent noise distribution with connected support set must be connected.
Revised: The output set produced by a continuous generator acting on a latent noise distribution with connected support set must be connected.

Original: 3. L ESSER OF T WO E VILS
Revised: 3. L ESSER OF T WO E VILS

Original: From the previous discussions, we know that the output of the generator must be connected, while diverse distributions must lie on supports consisting of several disconnected manifolds.
Revised: From the previous discussions, we know that the output of the generator must be connected, while diverse distributions must lie on supports consisting of several disconnected manifolds.

Original: If the generator does not suffer from mode dropping, then it must cover all the distinct modes in its output set.
Revised: If the generator does not suffer from mode dropping, then it must cover all the distinct modes in its output set.

Original: However, this implies the existence of parts in the generator’s output which connect these disconnected manifolds.
Revised: However, this implies the existence of parts in the generator’s output which connect these disconnected manifolds.

Original: We call such parts of the output space as tunnels.
Revised: We call such parts of the output space as tunnels.

Original: Since tunnels do not exist in the real distribution, they must correspond to unrealistic images.
Revised: Since tunnels do not exist in the real distribution, they must correspond to unrealistic images.

Original: We condense these ideas to the following result:
Revised: We condense these ideas to the following result:

Original: Result 4.
Revised: Result 4.

Original: A continuous generator, with inputs drawn from a connected set, must either suffer from signiﬁcant mode dropping or generate unrealistic images to some degree.
Revised: A continuous generator, with inputs drawn from a connected set, must either suffer from signiﬁcant mode dropping or generate unrealistic images to some degree.

Original: In the rest of the discussion, we assume that mode dropping is not a signiﬁcant problem.
Revised: In the rest of the discussion, we assume that mode dropping is not a signiﬁcant problem.

Original: This is a realistic assumption due to several heuristics and formulations that alleviate this problem (Salimans et al.
Revised: This is a realistic assumption due to several heuristics and formulations that alleviate this problem (Salimans et al.

Original: (2016), Che et al.
Revised: (2016), Che et al.

Original: (2016), Arjovsky et al.
Revised: (2016), Arjovsky et al.

Original: (2017), Gulrajani et al.
Revised: (2017), Gulrajani et al.

Original: (2017), Sun et al.
Revised: (2017), Sun et al.

Original: We concentrate on the problem of generation of unrealistic, distorted outputs.
Revised: We concentrate on the problem of generation of unrealistic, distorted outputs.

Original: 3. DCGAN S H AVE U NREALISTIC O UTPUTS IN H IGH P ROBABILITY D ENSITY R EGIONS

Original: While we have shown that unrealistic outputs must exist, a natural question is whether these outputs lie in regions of very low probability density, thus rendering our concerns meaningless in practice.

Original: In this section we assume the popular DCGAN (Radford et al., 2015) architecture, and show that unrealistic outputs lie in regions of high probability density.

Original: Empirical evidence strongly suggests that the mapping learnt by DCGANs (from latent noise to images) demonstrates interesting linear properties, e.g. given noise vectors z 1 corresponding to a smiling woman, z 2 corresponding to a frowning woman, the outputs corresponding to vectors linearly interpolated from z 1 to z 2 show a smooth transition from the smiling woman to the frowning woman.

Original: This experiment works because because the two images probably lie on the same manifold, and are connected through a sequence of intermediate realistic images.

Original: However, such experiments are likely to fail when the interpolation is done between images corresponding to sufﬁciently distinct modes, since the manifolds are disconnected.

Original: The intermediate images cannot possibly be realistic, while also representing a smooth transition (say between a bedroom and a face).

Original: This internal representation learnt by DCGANs, while interesting, poses a problem on datasets like

Original: ImageNet.

Original: All linear interpolations done on distinct modes are part of the tunnels, since they join disconnected manifolds and are not realistic.

Original: We now show that points on the line segments joining latent vectors of modes have high probability density.

Original: Lemma 1.

Original: Let z 1 , z 2 be points in R n .

Original: Let f ( x ) represent the probability density function of

Original: N ( µ, Σ ) , and z = θ z 1 + (1 − θ z 2 ) ( θ ∈ [0 , 1]) be any point on the line segment joining z 1 , z 2 .

Original: Then f ( z ) ≥ min( f ( z 1 ) , f ( z 2 )) .

Original: Proof.

Original: The PDF f ( x ) of a normal distribution is log-concave.

Original: Thus:

Original: [Equation]

Original: We now consider two sufﬁciently distinct modes in the generator’s output set with high probability density, with z 1 , z

Original: 2 being the corresponding latent vectors.

Original: From the above discussion, interpolations between z 1 and z 2 lead to unrealistic interpolations between the images.

Original: By Lemma 1, we know that these interpolations have probability density at least as high as one of the modes.

Original: We capture this discussion with the following result:

Original: Result 5.

Original: With the DCGAN architecture, interpolations between the noise vectors corresponding to sufﬁciently distinct, high probability modes map to unrealistic images which lie in regions of high probability density.

Original: 3. U NUSED M EASURE IN L IPSCHITZ C ONTINUOUS G ENERATORS
Revised: 3. U NUSED M EASURE IN L IPSCHITZ C ONTINUOUS G ENERATORS

Original: If we assume that the generator is K − Lipschitz (this happens with a variety of regularization terms added to the generator’s loss), i.e. for any two points z 1 , z 2 in the latent noise space we have
Revised: If we assume that the generator is K − Lipschitz (this happens with a variety of regularization terms added to the generator’s loss), i.e. for any two points z 1 , z 2 in the latent noise space we have

Original: [Equation]
Revised: [Equation]

Original: then the generator’s output must gradually shift from one manifold to another as we travel in Z space, since the slope is bounded.
Revised: then the generator’s output must gradually shift from one manifold to another as we travel in Z space, since the slope is bounded.

Original: The region of shift does not belong to any outputs in the real distribution.
Revised: The region of shift does not belong to any outputs in the real distribution.

Original: It is simple to see that some measure of the probability distribution Z is wasted on mapping these unwanted outputs.
Revised: It is simple to see that some measure of the probability distribution Z is wasted on mapping these unwanted outputs.

Original: To demonstrate this, we consider the simple case where P r consists of equal measure of samples of type A and type B, both of which are disconnected and highly distinct from each other.
Revised: To demonstrate this, we consider the simple case where P r consists of equal measure of samples of type A and type B, both of which are disconnected and highly distinct from each other.

Original: For simplicity, we assume that the latent noise z is drawn from a 1-D region [ − 1 , 1] , with uniform probability.
Revised: For simplicity, we assume that the latent noise z is drawn from a 1-D region [ − 1 , 1] , with uniform probability.

Original: Let the distance between set A and set B (deﬁned as inf a ∈ A,b ∈ B (cid:107) a − b (cid:107) ) be β ( > 0 due to high distinction between the sets).
Revised: Let the distance between set A and set B (deﬁned as inf a ∈ A,b ∈ B (cid:107) a − b (cid:107) ) be β ( > 0 due to high distinction between the sets).

Original: Let M A , M B be subsets of Z mapping to A and B respectively.
Revised: Let M A , M B be subsets of Z mapping to A and B respectively.

Original: For any arbitrary z 1 ∈ M A and z 2 ∈ M B , from the Lipschitz condition, we have:
Revised: For any arbitrary z 1 ∈ M A and z 2 ∈ M B , from the Lipschitz condition, we have:

Original: [Equation]
Revised: [Equation]

Original: Hence, the distance between any two points of M A and M B must be at least βK , as a result of which, the distance between the sets M A and M B is at least βK .
Revised: Hence, the distance between any two points of M A and M B must be at least βK , as a result of which, the distance between the sets M A and M B is at least βK .

Original: Clearly, in our hypothetical case, this results in a gap of at least βK wherever M A ends and M B begins.
Revised: Clearly, in our hypothetical case, this results in a gap of at least βK wherever M A ends and M B begins.

Original: Since we have assumed uniform distribution, a probability measure of β 2 K is lost to undesired outputs.
Revised: Since we have assumed uniform distribution, a probability measure of β 2 K is lost to undesired outputs.

Original: 3. C ONDITIONAL GAN S W ORK B ETTER ON D IVERSE D ATASETS
Revised: 3. C ONDITIONAL GAN S W ORK B ETTER ON D IVERSE D ATASETS

Original: It is well-known that conditional GANs (Mirza & Osindero (2014), Odena et al.
Revised: It is well-known that conditional GANs (Mirza & Osindero (2014), Odena et al.

Original: (2016)) are better at learning complex datasets.
Revised: (2016)) are better at learning complex datasets.

Original: The label information incorporated in the inputs of the generator plays a large role in making better outputs.
Revised: The label information incorporated in the inputs of the generator plays a large role in making better outputs.

Original: However, we also believe that the discontinuity introduced by the one-hot label representations contributes signiﬁcantly to improving performance.
Revised: However, we also believe that the discontinuity introduced by the one-hot label representations contributes signiﬁcantly to improving performance.

Original: More concretely, the input to the generator is a latent noise vector, along with the one-hot label representation.
Revised: More concretely, the input to the generator is a latent noise vector, along with the one-hot label representation.

Original: Hence, the input space is partitioned into n (number of classes) disconnected components due to the discrete nature of the one-hot labels.
Revised: Hence, the input space is partitioned into n (number of classes) disconnected components due to the discrete nature of the one-hot labels.

Original: This breaks the connectedness assumption of the input space, which was a crucial part of the problem’s cause.
Revised: This breaks the connectedness assumption of the input space, which was a crucial part of the problem’s cause.

Original: Note however that conditional GANs require labeled inputs, which may not be available for many datasets.
Revised: Note however that conditional GANs require labeled inputs, which may not be available for many datasets.

Original:  P ROPOSED S OLUTION
Revised:  P ROPOSED S OLUTIONS

Original: Central to the results in section 3 were the assumptions of continuity of the generator and the connectedness of the support set of the latent noise distribution.
Revised: Central to the results in section 3 were the assumptions of continuity of the generator and the connectedness of the support set of the latent noise distribution.

Original: Breaking either of these assumptions could lead to potential solutions.
Revised: Breaking either of these assumptions could lead to potential solutions.

Original: We now describea novel GAN formulation that breaks these assumptions while not requiring any labeled data.
Revised: We now describe novel GAN formulations that breaks these assumptions while not requiring any labeled data.

Original: 4. M ODEL
Revised: 4. M ODEL

Original: We create trainable discontinuities in the latent noise space by learning N different linear mappings { L 1 , L 2 . . .
Revised: We create trainable discontinuities in the latent noise space by learning N different linear mappings { L 1 , L 2 . . .

Original: L N } in the input layer of the generator.
Revised: L N } in the input layer of the generator.

Original: A noise vector z gets mapped to N vectors { y 1 =
Revised: A noise vector z gets mapped to N vectors { y 1 =

Original: L 1 ( z ) . . .
Revised: L 1 ( z ) . . .

Original:  L N ( z ) } by the input layer, and the rest of the processing remains the same as in standard generators.
Revised: y N = L N ( z ) } by the input layer, and the rest of the processing remains the same as in standard generators.

Original: We end with N outputs { g 1 =
Revised: We end with N outputs { g 1 =

Original: G ( y 1 ) , g 2 =
Revised: G ( y 1 ) , g 2 =

Original: G ( y 2 ) . . .
Revised: G ( y 2 ) . . .

Original: g N = G ( y N ) } .
Revised: g N = G ( y N ) } .

Original: Each of the linear layers L i maps the input latent noise space Z to a connected output set O i .
Revised: Each of the linear layers L i maps the input latent noise space Z to a connected output set O i .

Original: While
Revised: While

Original: O i is a connected set, the union of the N output sets ( O 1 ...
Revised: O i is a connected set, the union of the N output sets ( O 1 ...

Original: O N ) could potentially be disconnected (or each mapping could collapse to the same matrix, if required by the data).
Revised: O N ) could potentially be disconnected (or each mapping could collapse to the same matrix, if required by the data).

Original: This approach of partitioning the noise space can also be seen as training N different generators, with shared parameters, except for the input layer.
Revised: This approach of partitioning the noise space can also be seen as training N different generators, with shared parameters, except for the input layer.

Original: In this view, the generator function has become potentially discontinuous due to indexability of the outputs (i.e. choosing which output to take).
Revised: In this view, the generator function has become potentially discontinuous due to indexability of the outputs (i.e. choosing which output to take).

Original: In either view, we have broken one of the assumptions leading to the existence of tunnels.
Revised: In either view, we have broken one of the assumptions leading to the existence of tunnels.

Original: Finally, to facilitate the learning of distinct modes by each partition (or generator), we introduce a classiﬁer C that predicts which partition (or generator) created a given input.
Revised: Finally, to facilitate the learning of distinct modes by each partition (or generator), we introduce a classiﬁer C that predicts which partition (or generator) created a given input.

Original: We modify the GAN value function to suitably account for this change.
Revised: We modify the GAN value function to suitably account for this change.

Original: We would like to emphasize that this formulation is generic, and can be plugged in with different types of generators, discriminators, partition mappings, and classiﬁers.
Revised: We would like to emphasize that this formulation is generic, and can be plugged in with different types of generators, discriminators, partition mappings, and classiﬁers.

Original: Moreover, any improvements in GAN formulations can be independently incorporated into our setup.
Revised: Moreover, any improvements in GAN formulations can be independently incorporated into our setup.

Original: However, we do make speciﬁc design choices for the purpose of this paper, which we describe below.
Revised: However, we do make speciﬁc design choices for the purpose of this paper, which we describe below.

Original: To reduce the cost of training, we take C to be a linear mapping operating on the last hidden layer of the discriminator.
Revised: To reduce the cost of training, we take C to be a linear mapping operating on the last hidden layer of the discriminator.

Original: We believe that the discriminator extracts useful abstract features for judging the samples produced by the generator, and these can be effectively reused for partition classiﬁcation.
Revised: We believe that the discriminator extracts useful abstract features for judging the samples produced by the generator, and these can be effectively reused for partition classiﬁcation.

Original: 4. M ODIFIED L OSS F UNCTIONS
Revised: 4. M ODIFIED L OSS F UNCTIONS

Original: We add the partition classiﬁcation loss to the existing generator and discriminator losses from the 
Revised: We add the partition classiﬁcation loss to the existing generator and discriminator losses from the chosen base formulation.

Original: WGAN with Gradient Penalty formulation (Gulrajani et al., 2017).

Original: The new loss functions are:
Revised: The new loss functions are:

Original: [Equation]
Revised: [Equation]

Revised: L D , L G are the original loss functions for the discriminator and generator respectively, in the base

Revised: GAN formulation.

Original:  Here L c ( y, C ( x )) is the cross-entropy classiﬁcation loss for input image x , which was generated by partition y , and C ( x ) is the classiﬁer’s softmax output vector.
Revised: For vanilla GANs, [Equation] Here L c ( y, C ( x )) is the cross-entropy classiﬁcation loss for input image x , which was generated by partition y , and C ( x ) is the classiﬁer’s softmax output vector.

Original: α is a hyperparameter introduced to control the relative importance of generating good samples w.r.t encouraging diversity between the outputs of different partitions.
Revised: α is a hyperparameter introduced to control the relative importance of generating good samples w.r.t encouraging diversity between the outputs of different partitions.

Original: Finally, we must describe the exact meaning of P g in our formulation, since there are N generators.
Revised: Finally, we must describe the exact meaning of P g in our formulation, since there are N generators.

Original: We sample the generator at each training step uniformly, thus making P g
Revised: We sample the generator at each training step uniformly, thus making P g

Original: = 1 N (cid:80) Ni
Revised: = 1 N (cid:80) Ni

Original: =1 P g i , where P g i is the probability distribution induced by generator i .
Revised: =1 P g i , where P g i is the probability distribution induced by generator i .

Original: E XPERIMENTAL S ETUP 5. D ATASETS
Revised: 4. M IXTURE - OF -G AUSSIANS L ATENT D ISTRIBUTION

Revised: We now propose a simpler method for implementing a softer version of disconnectedness for the latent noise distribution. In this method, the noise vectors are drawn from a mixture-of-Gaussians with trainable means ( µ 1 , . . . , µ N ) and covariance matrices ( Diag ( σ 1 ) , . . .

Revised: , Diag ( σ N )) .

Revised: If required by the data, the means can move sufﬁciently far away from each other during training, leading to an almost disconnected distribution.

Revised: Each Gaussian is given equal probability weight, i.e., it is chosen uniformly at random.

Revised: Our implementation makes use of the reparameterization trick (Kingma & Welling (2013), Doersch (2016)).

Revised: We sample z ∼ N ( 0 , I ) , along with an index

Revised: i ∼ Uniform ( { 1 , . . . , N } ) .

Revised: We then use ˆ z = z (cid:12) σ i + µ i as the latent vector for generating the sample.

Revised: We let the gradients backpropagate to µ i and σ i , allowing for a learnable latent distribution.

Revised: Additionally, as mentioned before, we may use a classiﬁer to predict the Gaussian that generated a given sample, hence encouraging diversity between different Gaussians.

Revised:  E XPERIMENTAL S ETUP 5. D ATASETS

Original: We experiment with the image generation task on MNIST, celebA consisting of facial images,  and an artiﬁcial dataset called CelebRoom consisting of 100 , 000 images randomly sampled from celebA and 100 , 000 images randomly sampled from LSUN bedroom dataset.
Revised: We experiment with the image generation task on MNIST, celebA consisting of facial images, STL10, and an artiﬁcial dataset called CelebRoom consisting of 100 , 000 images randomly sampled from celebA and 100 , 000 images randomly sampled from LSUN bedroom dataset.

Original: CelebRoom was constructed with the explicit goal of including diverse modes, making it difﬁcult for GANs to train.
Revised: CelebRoom was constructed with the explicit goal of including diverse modes, making it difﬁcult for GANs to train.

Revised: As in Gulrajani et al.

Revised: , we present results on a toy dataset sampled from an equal mixturebivariate Gaussians.

Revised: The means are arranged uniformly in a circle with radius 2, and the covariance matrices are set to 0 .

Revised: 02 I .

Original: 5. I MPLEMENTATION
Revised: 5. I MPLEMENTATION

Original: We modiﬁed a popular TensorFlow (Abadi et al., 2015) implementation of the DCGAN architecturefor all experiments.
Revised: We modiﬁed a popular TensorFlow (Abadi et al., 2015) implementation of the WGAN-GP architecture 1 for all experiments.

Original: 5. H YPERPARAMETER T UNING
Revised: 5. H YPERPARAMETER T UNING

Original: α is tuned in powers of 10 from 10 − 3 to 10 3 , while N is ﬁxed for each dataset.
Revised: N is ﬁxed for each dataset.

Original: We use N =for MNIST, and N = 8 for celebA andCelebRoom.
Revised: We use N = 10 for MNIST, and N = 8 for celebA, STL-10, and

Original: We use 3 , 4 , 5 hidden layers for training on

Original: MNIST, celebA, and CelebRoom respecively.{+CelebRoom. α is ﬁxed to 1 . 
+}
Revised: {+CelebRoom. α is ﬁxed to 1 . 
+}

Original:  We also experiment between Adam (Kingma & Ba, 2014)and RMSProp (Tieleman & Hinton, 2012) for each dataset.
Revised: for all experiments with multiple partitions. We use the ADAM optimizer (Kingma & Ba, 2014) for all experiments.

Original: Again, we use the default hyperparameter values from DCGAN for Adam, and the same learning rate for RMSProp. For each dataset, we compared the samples obtainedby using three different setups - DCGAN architecture with vanilla formulation, DCGAN architecture with Wasserstein GAN and Gradient
Revised: For each dataset (except MNIST), we compare the samples obtained using the following setups: For MNIST and celebRoom, we also present results generated from the mixture-of-Gaussians latent distribution method.

Original: Penalty, and DCGAN architecture with Least Squares GAN (Mao et al., 2016) formulation.
Revised: [Table]

Original: We used the hyperparameters suggested by the original papers for each architecture.
Revised: We used the hyperparameters suggested by the original papers for each architecture.

Original: However, we observed that training quickly diverged for vanilla DCGAN and LSGAN, and hence we only report samples on Wasserstein GAN with Gradient Penalty.
Revised: The ResNet (He et al., 2016) architecture used has 4 residual blocks, as described in Gulrajani et al.

Original: We avoid intensive hyperparameter tuning in order to encourage discovery of methods with relatively high hyperparameter insensitivity.
Revised: We avoid intensive hyperparameter tuning in order to encourage discovery of methods with relatively high hyperparameter insensitivity.

Original: We compare our samples withthose generated by Wasserstein GANs with Gradient Penalty with DCGAN architecture. Where available, we show samples from the original paper.
Revised: We also tried training multiple partitions with the Least Squares GAN (Mao et al., 2016) formulation, but noticed that the training quickly diverged.

Original: The hyperparameters used are those suggested in the paper.
Revised: Hence we do not report these results here.

Original: 5. S AMPLES
Revised: 5. S AMPLES

Original: We present samples on the three datasets.
Revised: We present samples on the ﬁve datasets.

Original: For  each dataset, we show two types of samples: with and without the partition classiﬁer.
Revised: For samples generated using multiple partitions, each column represents the images produced by a single partition.  R ESULTS 6. 8 G AUSSIANS T OY D ATASET We observe that Vanilla GAN is unable to cover all Gaussians.

Original: Each column represents the images produced by a single partition.
Revised: As training proceeds, it shuttles between covering different subsets of Gaussians. WGAN-GP takes a long time to converge. It is ﬁnally able to cover all Gaussians, but not satisfactorily, with several samples lying between different means.

Original: R ESULTS 6. MNIST
Revised: We note that by reducing the gradient penalty signiﬁcantly, we are able to achieve faster convergence. However, we chose to use the recommended hyperparameters for all formulations. Vanilla GAN with partitions is able to cover all the modes, and converges quickly. 6. MNIST

Original: We observed a slight improvement in quality in the samples generated by our model when using the partition classiﬁer.
Revised: We observe a slight improvement in quality in the samples generated by our model when using the partition classiﬁer.

Original: The images produced with the classiﬁer formulation are more distinct, sharper, and do not contain spurious appendages.
Revised: The images produced with the classiﬁer formulation are more distinct, sharper, and do not contain spurious appendages.

Original: We expected each partition to produce a single type of digit, but this does not seem to be the case.
Revised: We expected each partition to produce a single type of digit, but this does not seem to be the case.

Original: Despite this, we do note that the partition classiﬁer’s loss was very close to zero upon successful training.
Revised: Despite this, we do note that the partition classiﬁer’s loss was very close to zero upon successful training.

Original: This signiﬁes that the partitions did learn to map distinct types of outputs, but not by digit label.
Revised: This signiﬁes that the partitions did learn to map distinct types of outputs, but not by digit label.

Original: This can be explained by the fact that labels are not the only way to classify digits.
Revised: This can be explained by the fact that labels are not the only way to classify digits.

Original: There are several combinations of digit style, tilt, width etc.
Revised: There are several combinations of digit style, tilt, width etc.

Original: that could be captured by the partitions
Revised: that could be captured by the partitions.

Original: . 6. CELEB A
Revised: 6. CELEB A

Original: We observed slight improvements in quality in the samples generated by our model over the WGAN + Gradient Penalty baseline.
Revised: We observed slight improvements in quality in the samples generated by our model over the WGAN + Gradient Penalty baseline with the DCGAN architecture.

Original: However, there is a signiﬁcant improvement in quality when we use a partition classiﬁer.
Revised: [Figure]

Original: Each partition seems to produce similar types of images, both with and without a partition classiﬁer formulation to encourage diversity.
Revised: Each partition seems to produce similar types of images, both with and without a partition classiﬁer formulation to encourage diversity.

Original: For instance,with the classiﬁer, the second partition (second column)  seems to capture the mode of smiling women, while the ﬁrst partition (ﬁrst column) seems to generate faces tilted to the right. 
Revised: For instance, the ﬁfth partition (ﬁfth column) in DCGAN architecture and WGAN-GP loss seems to capture the mode of smiling women, while the eigth partition (last column) seems to generate faces of men (often bespectacled). We also note that the ResNet architecture is unable to train successfully with partitions. We notice heavy mode collapse, and noisy outputs. 6. STL-

Revised: Our experiments show heavy conﬂation of modes in the outputs generated by all GAN formulations.

Original: 6. C ELEB R OOM
Revised: However, this problem is ameliorated to some extent with our partition GAN. In particular, we notice some partitions capturing distinctly well-formed modes. The DCGAN architecture with WGAN-GP loss with 8 partitions shows several such examples. The ﬁrst partition (ﬁrst column) seems to create images of vehicles, while the sixth partition (sixth column) seems to generate images of oceans and open skies (along with boats, ships, and airplanes).

Revised: Similar clustering can also be observed with the ResNet architecture with WGAN-GP loss, where partition four (column four) generates images of birds and planes, while the sixth partition (column six) generates images of animals.

Revised: 6. C ELEB R OOM

Original: We observe signiﬁcant conﬂation of bedrooms and facial images in this dataset with the baseline model.
Revised: We observe signiﬁcant conﬂation of bedrooms and facial images in this dataset with the baseline model.

Original: Our model alleviates this problem to some extent, but does not solve it completely.
Revised: Our model alleviates this problem to some extent, but does not solve it completely.

Original: However, we see that each partition does either  create faces or bedrooms.
Revised: However, we see that each partition does either primarily create faces or bedrooms.

Original: Using a partition classiﬁer improves the sample quality slightly.

Original: We would like to note that if a particular partition generates bad quality samples, it might be due to the inherent difﬁculty in creating that portion of the real distribution.
Revised: We would like to note that if a particular partition generates bad quality samples, it might be due to the inherent difﬁculty in creating that portion of the real distribution.

Original: In such cases, we can drop the outputs from that partition at the cost of not capturing 1 N of the distribution.
Revised: In such cases, we can drop the outputs from that partition at the cost of not capturing 1 N of the distribution.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We highlighted a major problem in training GANs on complex image datasets and introduced theoretical analysis for the problem of generation of unrealistic, conﬂated images in such cases.
Revised: We highlighted a major problem in training GANs on complex image datasets and introduced theoretical analysis for the problem of generation of unrealistic, conﬂated images in such cases.

Original: We proposed the addition of discontinuity in latent noise space of the generator for covering disjoint and diverse modes of the data distribution, and augmented the loss functions to encourage diversity.
Revised: We proposed the addition of discontinuity in latent noise space of the generator for covering disjoint and diverse modes of the data distribution, and augmented the loss functions to encourage diversity.

Original: We showed improvements over existing models without much hyperparameter tuning.
Revised: We showed improvements over existing models without much hyperparameter tuning.

Original: In future, we hope to perform an extensive exploration of the search space to obtain a set of hyperparameters along with better methods to introduce discontinuities in the generator that perform well on a variety of datasets, while signiﬁcantly improving image quality.
Revised: In future, we hope to perform an extensive exploration of the search space to obtain a set of hyperparameters along with better methods to introduce discontinuities in the generator that perform well on a variety of datasets, while signiﬁcantly improving image quality