Original: AIM: A DVERSARIAL I NFERENCE BY M ATCHING P RIORS AND C ONDITIONALS
Revised: AIM: A DVERSARIAL I NFERENCE BY M ATCHING P RIORS AND C ONDITIONALS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Effective inference for a generative adversarial model remains an important and challenging problem.
Revised: Effective inference for a generative adversarial model remains an important and challenging problem.

Original: We propose a novel approach, Adversarial Inference by
Revised: We propose a novel approach, Adversarial Inference by

Original: Matching priors and conditionals (AIM), which explicitly matches prior and conditional distributions in both data and code spaces, and puts a direct constraint on the dependency structure of the generative model.
Revised: Matching priors and conditionals (AIM), which explicitly matches prior and conditional distributions in both data and code spaces, and puts a direct constraint on the dependency structure of the generative model.

Original: We derive an equivalent form of the prior and conditional matching objective that can be optimized efﬁciently without any parametric assumption on the data.
Revised: We derive an equivalent form of the prior and conditional matching objective that can be optimized efﬁciently without any parametric assumption on the data.

Original: We validate the effectiveness of AIM on the MNIST, CIFAR-10, and CelebA datasets by conducting quantitative and qualitative evaluations.
Revised: We validate the effectiveness of AIM on the MNIST, CIFAR-10, and CelebA datasets by conducting quantitative and qualitative evaluations.

Original: Results demonstrate that AIM signiﬁcantly improves both reconstruction and generation as compared to other adversarial inference models
Revised: Results demonstrate that AIM signiﬁcantly improves both reconstruction and generation as compared to other adversarial inference models

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep directed generative models like variational autoencoder (VAE) (Kingma and Welling, 2013;
Revised: Deep directed generative models like variational autoencoder (VAE) (Kingma and Welling, 2013;

Original: Rezende et al., 2014) and generative adversarial network (GAN) (Goodfellow et al., 2014) have been proved to be powerful for modeling complex high-dimensional distributions.
Revised: Rezende et al., 2014) and generative adversarial network (GAN) (Goodfellow et al., 2014) have been proved to be powerful for modeling complex high-dimensional distributions.

Original: While both VAE and
Revised: While both VAE and

Original: GAN can learn to generate realistic images, their underlying mechanisms are fundamentally different.
Revised: GAN can learn to generate realistic images, their underlying mechanisms are fundamentally different.

Original: VAE maps the data into low-dimensional codes using an encoder, and then reconstructs the original data by a decoder.
Revised: VAE maps the data into low-dimensional codes using an encoder, and then reconstructs the original data by a decoder.

Original: This allows it to perform both generation and inference.
Revised: This allows it to perform both generation and inference.

Original: GAN, on the other hand, trains a generator and a discriminator adversarially.
Revised: GAN, on the other hand, trains a generator and a discriminator adversarially.

Original: The generator learns to fool the discriminator by mapping low-dimensional noise vectors to the data space; at the same time, the discriminator evolves to detect the generated fake samples from the true ones.
Revised: The generator learns to fool the discriminator by mapping low-dimensional noise vectors to the data space; at the same time, the discriminator evolves to detect the generated fake samples from the true ones.

Original: These two methods have complementary strengths and weaknesses.
Revised: These two methods have complementary strengths and weaknesses.

Original: VAE can learn a bidirectional mapping between data and code spaces, but relies on over-simpliﬁed parametric assumptions on the complex data distribution, thereby causing it to generate blurry images (Donahue et al., 2016; Goodfellow et al., 2014; Larsen et al., 2015).
Revised: VAE can learn a bidirectional mapping between data and code spaces, but relies on over-simpliﬁed parametric assumptions on the complex data distribution, thereby causing it to generate blurry images (Donahue et al., 2016; Goodfellow et al., 2014; Larsen et al., 2015).

Original: GAN generates more realistic samples than VAE (Radford et al., 2015; Larsen et al., 2015) because the adversarial regime allows it to learn more complex distributions.
Revised: GAN generates more realistic samples than VAE (Radford et al., 2015; Larsen et al., 2015) because the adversarial regime allows it to learn more complex distributions.

Original: However, note that GAN only learns a unidirectional mapping for data generation, and does not allow inferring the latent codes from given samples.
Revised: However, note that GAN only learns a unidirectional mapping for data generation, and does not allow inferring the latent codes from given samples.

Original: This is limiting because the ability of inference is very crucial for several downstream applications, such as classiﬁcation, clustering, similarity search, and interpretation.
Revised: This is limiting because the ability of inference is very crucial for several downstream applications, such as classiﬁcation, clustering, similarity search, and interpretation.

Original: Furthermore,
Revised: Furthermore,

Original: GAN also suffers from the model collapse problem (Che et al., 2016; Salimans et al., 2016) – many modes of the data distribution are not represented in the generated samples.
Revised: GAN also suffers from the model collapse problem (Che et al., 2016; Salimans et al., 2016) – many modes of the data distribution are not represented in the generated samples.

Original: Therefore, one may wonder on whether it is possible to develop a generative model that enjoys the strengths of both GAN and VAE without their inherent weaknesses.
Revised: Therefore, one may wonder on whether it is possible to develop a generative model that enjoys the strengths of both GAN and VAE without their inherent weaknesses.

Original: Such model should be able to generate high-quality samples as good as GAN, have an inference mechanism as effective as VAE, and also avoid the mode collapse issue.
Revised: Such model should be able to generate high-quality samples as good as GAN, have an inference mechanism as effective as VAE, and also avoid the mode collapse issue.

Original: Many recent efforts have been devoted to combining VAE with adversarial discriminator(s)
Revised: Many recent efforts have been devoted to combining VAE with adversarial discriminator(s)

Original: (Brock et al., 2016; Che et al., 2016; Larsen et al., 2015; Makhzani et al., 2015).
Revised: (Brock et al., 2016; Che et al., 2016; Larsen et al., 2015; Makhzani et al., 2015; Mescheder et al., 2017).

Original: However, VAE-GAN hybrids tend to manifest a compromise of the strengths and weaknesses of both the approaches.
Revised: However, VAE-GAN hybrids tend to manifest a compromise of the strengths and weaknesses of both the approaches.

Original: The main reason is that all of them retain the autoencoder structure, which requires an explicit metric to measure the data reconstruction and assumes over-simpliﬁed parametric data distributions.
Revised: The main reason is that all of them retain the autoencoder structure, which requires an explicit metric to measure the data reconstruction and assumes over-simpliﬁed parametric data distributions.

Original: To overcome such limitations, adversarially learned inference (ALI) (Donahue et al., 2016; Dumoulin et al., 2016) was recently proposed, wherein the discriminator is trained on the joint distribution of data and latent codes.
Revised: To overcome such limitations, adversarially learned inference (ALI) (Donahue et al., 2016; Dumoulin et al., 2016) was recently proposed, wherein the discriminator is trained on the joint distribution of data and latent codes.

Original: In this way, under a perfect discriminator, one can successfully match joint distributions of the decoder and encoder, thereby, performing inference by sampling from the encoder’s conditional distribution that also matches the decoder’s posterior.
Revised: In this way, under a perfect discriminator, one can successfully match joint distributions of the decoder and encoder, thereby, performing inference by sampling from the encoder’s conditional distribution that also matches the decoder’s posterior.

Original: However, in practice the equilibrium of the jointly adversarial game is hard to approach as the dependency structure between data and codes is not explicitly speciﬁed.
Revised: However, in practice the equilibrium of the jointly adversarial game is hard to approach as the dependency structure between data and codes is not explicitly speciﬁed.

Original: ALI’s reconstructions of samples are thus not always faithful (Dumoulin et al., 2016; Li et al., 2017) implying that its inference mechanism is not always effective.
Revised: ALI’s reconstructions of samples are thus not always faithful (Dumoulin et al., 2016; Li et al., 2017) implying that its inference mechanism is not always effective.

Original: To overcome the aforementioned issues, in this paper, we propose a novel approach, Adversarial
Revised: To overcome the aforementioned issues, in this paper, we propose a novel approach, Adversarial

Original: Inference by Matching priors and conditionals (AIM), that integrates efﬁcient inference to GAN and overcomes the limitations of prior approaches.
Revised: Inference by Matching priors and conditionals (AIM), that integrates efﬁcient inference to GAN and overcomes the limitations of prior approaches.

Original: The approach keeps the structure simple, involving only one generator, one encoder, and one discriminator.
Revised: The approach keeps the structure simple, involving only one generator, one encoder, and one discriminator.

Original: Furthermore, AIM’s objective is directly derived from our goal of matching both prior and conditional distributions of the generator and encoder, instead of a heuristic combination with l k norm regularization.
Revised: Furthermore, AIM’s objective is directly derived from our goal of matching both prior and conditional distributions of the generator and encoder, instead of a heuristic combination with l k norm regularization.

Original: Compared to regular
Revised: Compared to regular

Original: GANs, AIM has the ability to conduct inference, and also does not suffer from the mode collapse problem.
Revised: GANs, AIM has the ability to conduct inference, and also does not suffer from the mode collapse problem.

Original: Moreover, AIM also abandons the unrealistic parametric assumption on the conditional data distribution, and does not require any reconstruction in the data space.
Revised: Moreover, AIM also abandons the unrealistic parametric assumption on the conditional data distribution, and does not require any reconstruction in the data space.

Original: This is fundamentally different from VAE or VAE-GAN hybrids in which the l k norm is used to measure the data reconstruction.
Revised: This is fundamentally different from VAE or VAE-GAN hybrids in which the l k norm is used to measure the data reconstruction.

Original: The usage of simple data-ﬁtting metrics on the complex data distribution leads to worse generation performance.
Revised: The usage of simple data-ﬁtting metrics on the complex data distribution leads to worse generation performance.

Original: Different from ALI, AIM decomposes the hard problem of matching the joint distributions into two sub-tasks – explicitly matching the priors on the latent codes and the conditionals on the data.
Revised: Different from ALI, AIM decomposes the hard problem of matching the joint distributions into two sub-tasks – explicitly matching the priors on the latent codes and the conditionals on the data.

Original: As a consequence of more restrictive constraint, it achieves better generation and more faithful reconstruction than ALI.
Revised: As a consequence of more restrictive constraint, it achieves better generation and more faithful reconstruction than ALI.

Original: Last but not the least, note that GAN variations with inference mechanism usually achieve worse generation performance as compared to regular GANs.
Revised: Last but not the least, note that GAN variations with inference mechanism usually achieve worse generation performance as compared to regular GANs.

Original: However, as demonstrated in the experiments, AIM not only performs best on inference, but also improves the generation performance over GAN.
Revised: However, as demonstrated in the experiments, AIM not only performs best on inference, but also improves the generation performance over GAN.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The most straightforward way to learn an inference mechanism is to learn the inverse mapping of
Revised: The most straightforward way to learn an inference mechanism is to learn the inverse mapping of

Original: GAN’s generator post-hoc (Zhu et al., 2016).
Revised: GAN’s generator post-hoc (Zhu et al., 2016).

Original: However, since its training process is the same as
Revised: However, since its training process is the same as

Original: GAN, it still suffers from mode collapse problem.
Revised: GAN, it still suffers from mode collapse problem.

Original: AGE (Ulyanov et al., 2017) encourages encoder and generator to be reciprocal by simultaneously minimizing an l 1 reconstruction error in the data space and an l 2 error in the code space.
Revised: AGE (Ulyanov et al., 2017) encourages encoder and generator to be reciprocal by simultaneously minimizing an l 1 reconstruction error in the data space and an l 2 error in the code space.

Original: This is closely related to the cycle-consistency criterion (Zhu et al., 2017; Kim et al., 2017;
Revised: This is closely related to the cycle-consistency criterion (Zhu et al., 2017; Kim et al., 2017;

Original: Yi et al., 2017; Li et al., 2017).
Revised: Yi et al., 2017; Li et al., 2017).

Original: Although the pairwise reconstruction errors help reduce mode collapse, the data reconstruction is still measured by l 1 or l 2 norm, which brings the same problem of VAE and VAE-GAN hybrids.
Revised: Although the pairwise reconstruction errors help reduce mode collapse, the data reconstruction is still measured by l 1 or l 2 norm, which brings the same problem of VAE and VAE-GAN hybrids.

Original: InfoGAN (Chen et al., 2016) minimizes the mutual information between a subset c of the latent code and the generated samples, and hence can only do partial inference on c .
Revised: InfoGAN (Chen et al., 2016) minimizes the mutual information between a subset c of the latent code and the generated samples, and hence can only do partial inference on c .

Revised: Different from the heuristic combination of VAE and GANs, Mescheder et al.

Revised: (2017) theoretically derived an adversarial game to replace the KL-divergence term in the variational lower bound (also called ELBO), and gives the new method, adversarial variational Bayes (AVB), much more ﬂexibility in its dependence on latent z .

Revised: However, the reconstruction term on x still exists and so is the parametric assumption on the conditional data distribution, leading to the blurriness in their reconstructed and generated samples.

Original: ALI (Dumoulin et al., 2016; Donahue et al., 2016) is an elegant approach to bring inference mechanism into adversarial learning without assuming parametric distribution on the data.
Revised: ALI (Dumoulin et al., 2016; Donahue et al., 2016) is an elegant approach to bring inference mechanism into adversarial learning without assuming parametric distribution on the data.

Original: Different from our work, it directly plays an adversarial game to match the joint distributions of the decoder and encoder.
Revised: Different from our work, it directly plays an adversarial game to match the joint distributions of the decoder and encoder.

Original: But in practice, ALI’s reconstructions are not necessarily faithful because the dependency structures within the two joint distributions are not speciﬁed (Li et al., 2017).
Revised: But in practice, ALI’s reconstructions are not necessarily faithful because the dependency structures within the two joint distributions are not speciﬁed (Li et al., 2017).

Original: ALICE (Li et al., 2017) tries to solve this problem by regularizing ALI using an extra conditional entropy constraint on the data.
Revised: ALICE (Li et al., 2017) tries to solve this problem by regularizing ALI using an extra conditional entropy constraint on the data.

Original: The conditional entropy is either explicitly measured by l k norm, or implicitly learned by adversarial training.
Revised: The conditional entropy is either explicitly measured by l k norm, or implicitly learned by adversarial training.

Original: However, when the data distribution becomes complicated (e.g. CIFAR-10), the l k metric may lead to blurry reconstructions and the adversarial training is hard to achieve (Li et al., 2017).
Revised: However, when the data distribution becomes complicated (e.g. CIFAR-10), the l k metric may lead to blurry reconstructions and the adversarial training is hard to achieve (Li et al., 2017).

Original: Compared with ALI and ALICE, our method is proven to minimize the KL-divergence between both priors and conditionals of generator and encoder, and can provide consistent effective inference even on complicated distribution (see Section 5.1).
Revised: Compared with ALI and ALICE, our method is proven to minimize the KL-divergence between both priors and conditionals of generator and encoder, and can provide consistent effective inference even on complicated distribution (see Section 5.1).

Original: B ACKGROUND
Revised: Srivastava et al. (2017) proposed VEEGAN to tackle the mode collapse issue of GANs by adding an implicit variatinoal learning on the latent z . To our best knowledge, this is by far the only approach that is also reconstructing z . Different from VAEs, VEEGAN autoencodes the latent variable or noise z .

Original: [Figure]
Revised: By doing so, it enforces the generator not to collapse the mappings of z to a single mode, because otherwise, the encoder will not be able to recover all the noise z . The details of their model can be summarized as ALI regularized by an extra reconstruction of latent z . Therefore, VEEGAN is similar to ALICE in the sense that they are both adversarial games on the joint distribution with an extra regularization on either data or latent reconstruction. Our model AIM instead only plays the adversarial game on the marginal data distribution, and reconstructs the latent z by maximizing its log-likelihood under the latent posterior distribution.

Original:  x
Revised:  B ACKGROUND

Original: [Figure]
Revised: [Figure]

Original: We consider the generative model p θ ( z ) p θ ( x | z ) , where a latent variable z ( i ) is ﬁrst generated from the prior distribution p θ ( z ) , and then the data x ( i ) is sampled from the conditional distribution p θ ( x | z ) .
Revised: We consider the generative model p θ ( z ) p θ ( x | z ) , where a latent variable z ( i ) is ﬁrst generated from the prior distribution p θ ( z ) , and then the data x ( i ) is sampled from the conditional distribution p θ ( x | z ) .

Original: The parameter θ stands for the ground truth parameter of the distribution.
Revised: The parameter θ stands for the ground truth parameter of the distribution.

Original: The prior p θ ( z ) is always assumed to be a simple parametric distribution (e.g. N ( 0 , I ) ), but the generative conditional p θ ( x | z ) is always very complex and not known to us.
Revised: The prior p θ ( z ) is always assumed to be a simple parametric distribution (e.g. N ( 0 , I ) ), but the generative conditional p θ ( x | z ) is always very complex and not known to us.

Original: Moreover, the posterior p θ ( z | x ) is intractable but stands for an important inference procedure: given a data x ( i ) , it allows us to infer its latent variable z ( i ) .
Revised: Moreover, the posterior p θ ( z | x ) is intractable but stands for an important inference procedure: given a data x ( i ) , it allows us to infer its latent variable z ( i ) .

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In our method, we will model the generating process by a neural network called generator , and the inference process by another neural network encoder .
Revised: In our method, we will model the generating process by a neural network called generator , and the inference process by another neural network encoder .

Original: Consider the following two distributions of the generator and encoder, and their corresponding sampling procedures:
Revised: Consider the following two distributions of the generator and encoder, and their corresponding sampling procedures:

Original: • the generator distribution: p ( z ) p ( x | z ) ; z ∼ p ( z ) , x ∼ p ( x | z ) .
Revised: • the generator distribution: p ( z ) p ( x | z ) ; z ∼ p ( z ) , x ∼ p ( x | z ) .

Original: • the encoder distribution: q ( x ) q ( z | x ) ; x ∼ q ( x ) , z ∼ q ( z | x ) .
Revised: • the encoder distribution: q ( x ) q ( z | x ) ; x ∼ q ( x ) , z ∼ q ( z | x ) .

Original: The generator’s conditional p ( x | z ) approximates the generating distribution p θ ( x | z ) .
Revised: The generator’s conditional p ( x | z ) approximates the generating distribution p θ ( x | z ) .

Original: The encoder’s conditional q ( z | x ) approximates the posterior distribution p θ ( z | x ) , which is what we need for inference.
Revised: The encoder’s conditional q ( z | x ) approximates the posterior distribution p θ ( z | x ) , which is what we need for inference.

Original: The marginal distribution q ( x ) stands for the empirical data distribution, and the other marginal p ( z ) is taken to be p θ ( z ) , which is always a known distribution like standard Gaussian.
Revised: The marginal distribution q ( x ) stands for the empirical data distribution, and the other marginal p ( z ) is taken to be p θ ( z ) , which is always a known distribution like standard Gaussian.

Original: 4. M ATCHING PRIORS AND CONDITIONALS
Revised: 4. M ATCHING PRIORS AND CONDITIONALS

Original: The ultimate goal is to match the joint distributions, p ( x , z ) and q ( x , z ) .
Revised: The ultimate goal is to match the joint distributions, p ( x , z ) and q ( x , z ) .

Original: If this is achieved, we are guaranteed that all marginals match and all conditionals match as well.
Revised: If this is achieved, we are guaranteed that all marginals match and all conditionals match as well.

Original: In particular, the conditional q ( z | x ) matches the posterior p ( z | x ) .
Revised: In particular, the conditional q ( z | x ) matches the posterior p ( z | x ) .

Original: We propose to decompose this goal into two sub-tasks – matching the priors p ( z ) and q ( z ) , and matching the conditionals p ( x | z ) and q ( x | z ) .
Revised: We propose to decompose this goal into two sub-tasks – matching the priors p ( z ) and q ( z ) , and matching the conditionals p ( x | z ) and q ( x | z ) .

Original: There are two advantages.
Revised: There are two advantages.

Original: Firstly, we explicitly deﬁne the dependency structure z → x .
Revised: Firstly, we explicitly deﬁne the dependency structure z → x .

Original: Secondly, the explicit constraints on both priors and conditionals are stronger than merely one constraint on the joint distributions.
Revised: Secondly, the explicit constraints on both priors and conditionals are stronger than merely one constraint on the joint distributions.

Original: In order to match the distributions, we will minimize the KL-divergence between the priors and between the conditionals:
Revised: In order to match the distributions, we will minimize the KL-divergence between the priors and between the conditionals:

Original: [Equation]
Revised: [Equation]

Original: By the properties of KL-divergence, when the minimum of equation 1 is attained, we have both p ( z ) =
Revised: By the properties of KL-divergence, when the minimum of equation 1 is attained, we have both p ( z ) =

Original: q ( z ) and p ( x | z ) =
Revised: q ( z ) and p ( x | z ) =

Original: q ( x | z ) , for all x and z .
Revised: q ( x | z ) , for all x and z .

Original: 4. O BJECTIVE FUNCTION
Revised: 4. O BJECTIVE FUNCTION

Original: The objective equation 1 cannot be directly optimized because both q ( z ) and q ( x | z ) are impossible to sample from, as the ﬂow in the encoder is from x to z .
Revised: The objective equation 1 cannot be directly optimized because both q ( z ) and q ( x | z ) are impossible to sample from, as the ﬂow in the encoder is from x to z .

Original: Therefore, we rewrite equation 1 as the combination of a KL-divergence term and a reconstruction term, so that both of them only contain distributions that can be sampled from or directly evaluated.
Revised: Therefore, we rewrite equation 1 as the combination of a KL-divergence term and a reconstruction term, so that both of them only contain distributions that can be sampled from or directly evaluated.

Original: Firstly, for any ﬁxed z , the KL-divergence between the conditionals, D KL ( p ( x | z ) || q ( x | z )) , can be decomposed as:
Revised: Firstly, for any ﬁxed z , the KL-divergence between the conditionals, D KL ( p ( x | z ) || q ( x | z )) , can be decomposed as:

Original: [Equation]
Revised: [Equation]

Original: Then by adding (log p ( z ) − log p ( z )) − log q ( z ) to both sides of equation 2, we have
Revised: Then by adding (log p ( z ) − log p ( z )) − log q ( z ) to both sides of equation 2, we have

Original: D KL ( p ( x | z ) || q ( x | z )) +(log p ( z ) − log q ( z )) − log p ( z ) = D KL ( p ( x | z ) || q ( x )) − E p ( x | z ) [log q ( z | x )] .
Revised: [Equation]

Original: Now taking expectation with respect to p ( z ) on both sides, we get
Revised: Now taking expectation with respect to p ( z ) on both sides, we get

Original: [Equation]
Revised: [Equation]

Original: Note that the term E p ( z ) log p ( z ) is a constant because the prior p ( z ) is a ﬁxed parametric distribution.
Revised: Note that the term E p ( z ) log p ( z ) is a constant because the prior p ( z ) is a ﬁxed parametric distribution.

Original: For example, when z ∼ N ( 0 , I d ) , we have E p ( z ) log p ( z ) = − d
Revised: For example, when z ∼ N ( 0 , I d ) , we have E p ( z ) log p ( z ) = − d

Original: 1 + log(2 π )) / 2 .
Revised: 1 + log(2 π )) / 2 .

Original: Therefore, minimizing the objective equation 1 is now transformed to minimizing the new objective equation 3.
Revised: Therefore, minimizing the objective equation 1 is now transformed to minimizing the new objective equation 3.

Original: Intuitively, term (I) measures the difference between generated and real samples, and term (II) measures the reconstruction of the latent codes.
Revised: Intuitively, term (I) measures the difference between generated and real samples, and term (II) measures the reconstruction of the latent codes.

Original: We summarize the above procedure as a proposition.
Revised: We summarize the above procedure as a proposition.

Original: Proposition 1 The objective function
Revised: Proposition 1 The objective function

Original: [Equation]
Revised: [Equation]

Original: is minimized when p ( z ) =
Revised: is minimized when p ( z ) =

Original: q ( z ) and p ( x | z ) =
Revised: q ( z ) and p ( x | z ) =

Original: q ( x | z ) for all z , x .
Revised: q ( x | z ) for all z , x .

Original: And hence, at the minimum, the joint distributions p ( x , z )
Revised: And hence, at the minimum, the joint distributions p ( x , z )

Original: =
Revised: =

Original: q ( x , z ) .
Revised: q ( x , z ) .

Original: 4. AIM FRAMEWORK
Revised: 4. R ELATION TO VAE

Revised: The VAE (Kingma and Welling, 2013) method, using our notations in this paper, actually depends on the following identity:

Revised: [Equation]

Revised: Then because of the non-negativity of KL-divergence, we have

Revised: [Equation]

Revised: and hence maximizing the log-likelihood of the observations can be transferred to maximizing the evidence lower bound (ELBO).

Revised: But taking a closer look at equation 4 and comparing it to equation 2, we notice that equation 4 is a decomposition of the KL-divergence between two conditionals, q ( z | x ) and p ( z | x ) .

Revised: Therefore, we can follow the same approach after equation 2 and get the following identity:

Revised: [Equation]

Revised: Since the marginal q ( x ) stands for the empirical data distribution, the right hand side of equationis the empirical expectation of the negative ELBO, which is what VAE tries to minimize.

Revised: We then conclude from equation 5 that VAE performs marginal distribution matching in the data space and conditional distribution matching in the latent space.

Revised: This distribution matching of VAE is also observed by Rosca et al.

Revised: However, the marginal distributions in the data space are very complex, and the direction x

Revised: → z in the conditional distributions in the latent space is actually opposite to the generating process z

Revised: → x .

Revised: Hence, in order to match these distributions, VAE’s objective has a reconstruction term II vae on x , and a regularization term I vae on latent z .

Revised: But to evaluate both terms, we need to make parametric assumptions on both conditionals q ( z | x ) and p ( x | z ) .

Revised: The assumption on I vae can be loosed using

Revised: GANs (Makhzani et al., 2015), but the assumption on II vae is critical and limits the performance of

Revised: VAE-GAN hybrids.

Revised: Our model, AIM, instead performs marginal distribution matching in the latent space and conditional distribution matching in the data space.

Revised: From equation 3, since the term I will be replaced with an adversarial game (see Section 4.4), the only assumption we need to make is on term II, that is, on the conditional q ( z | x ) .

Revised: And our model is very ﬂexible in its dependence on z .

Revised: This assumption is much weaker than that on p ( x | z ) and does not lead to the problems of VAE or VAE-GANs (e.g. blurriness).

Revised: 4. AIM FRAMEWORK

Original: The KL-divergence part (I) can be replaced by an adversarial game using the f -divergence theory (Nowozin et al., 2016).
Revised: The KL-divergence part (I) can be replaced by an adversarial game using the f -divergence theory (Nowozin et al., 2016).

Original: The reconstruction term (II) is a log-likelihood and can be simply evaluated if we assume a parametric q ( z | x ) .
Revised: The reconstruction term (II) is a log-likelihood and can be simply evaluated if we assume a parametric q ( z | x ) .

Original: Therefore, our framework only requires exactly one generator G , one discriminator D , and one encoder E .
Revised: Therefore, our framework only requires exactly one generator G , one discriminator D , and one encoder E .

Original: We will now discuss how to play the adversarial game and measure the reconstruction in details.
Revised: We will now discuss how to play the adversarial game and measure the reconstruction in details.

Original: Adversarial game Because we do not want to make any parametric assumption on the distribution p ( x | z ) , an adversarial game will be played to distinguish p ( x | z ) from q ( x ) .
Revised: Adversarial game Because we do not want to make any parametric assumption on the distribution p ( x | z ) , an adversarial game will be played to distinguish p ( x | z ) from q ( x ) .

Original: By the theory of f -GAN (Nowozin et al., 2016), we then construct an adversarial game with the value function
Revised: By the theory of f -GAN (Nowozin et al., 2016), we then construct an adversarial game with the value function

Original: [Equation]
Revised: [Equation]

Original: Under the perfect discriminator, ﬁnding the optimal generator is then equivalent to minimizing the
Revised: Under the perfect discriminator, ﬁnding the optimal generator is then equivalent to minimizing the

Original: KL-divergence.
Revised: KL-divergence.

Original: The activation function for the discriminator in equation 4 is just the identity mapping instead of the sigmoid function in the original GAN.
Revised: The activation function for the discriminator in equation 6 is just the identity mapping instead of the sigmoid function in the original GAN.

Original: But just like in the original GAN, the generator of equation 4 also suffers from the gradient vanishing problem (Goodfellow, 2016).
Revised: But just like in the original GAN, the generator of equation 6 also suffers from the gradient vanishing problem (Goodfellow, 2016).

Original: Since the original
Revised: Since the original

Original: GAN is well studied and sufﬁces for our purpose, we will instead use its value function:
Revised: GAN is well studied and sufﬁces for our purpose, we will instead use its value function:

Original: [Equation]
Revised: [Equation]

Original: And as suggested in Goodfellow (2016), in order to mitigate the generator’s gradient vanishing problem, we minimize another value function − E z ∼ p
Revised: And as suggested in Goodfellow (2016), in order to mitigate the generator’s gradient vanishing problem, we minimize another value function − E z ∼ p

Original: ( z ) [log D ( G ( z ))] for the generator.
Revised: ( z ) [log D ( G ( z ))] for the generator.

Original: Reconstruction Because of the simplicity of the distribution on z , we make a reasonable parametric assumption on q ( z | x ) so that the log-likelihood can be explicitly calculated.
Revised: Reconstruction Because of the simplicity of the distribution on z , we make a reasonable parametric assumption on q ( z | x ) so that the log-likelihood can be explicitly calculated.

Original: In this paper we will assume z | x ∼ N ( µ ( x ) , σ 2 ( x ) I ) , and hence
Revised: In this paper we will assume z | x ∼ N ( µ ( x ) , σ 2 ( x ) I ) , and hence

Original: [Equation]
Revised: [Equation]

Original: where d is the dimension of z .
Revised: where d is the dimension of z .

Original: In this case, the encoder network only needs to output two vectors, µ ( x ) and
Revised: In this case, the encoder network only needs to output two vectors, µ ( x ) and

Original: σ 2 ( x ) , that is, E ( x ) =
Revised: σ 2 ( x ) , that is, E ( x ) =

Original: ( µ ( x ) , σ 2 ( x )) .
Revised: ( µ ( x ) , σ 2 ( x )) .

Original: Then we can compute the approximate negative posterior log-likelihood by plugging E ( x ) into equation 6, i.e. L ( z , E ( x )) .
Revised: Then we can compute the approximate negative posterior log-likelihood by plugging E ( x ) into equation 8, i.e. L ( z , E ( x )) .

Original: To summarize, our ﬁnal optimization problem is
Revised: To summarize, our ﬁnal optimization problem is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Here, if we use V ( G, D ) in equation 4 derived from f -GAN, then λ
Revised: Here, if we use V ( G, D ) in equation 6 derived from f -GAN, then λ

Original: = 1 .
Revised: = 1 .

Original: If we use V ( G, D ) in equation 5, then λ needs to be set so that two parts of equation 7 are in the same scale.
Revised: If we use V ( G, D ) in equation 7, then λ needs to be set so that two parts of equation 9 are in the same scale.

Original: We will discuss this in detail in Section 5.
Revised: We will discuss this in detail in Section 5.

Original: 4. T RAINING AND INFERENCE PROCEDURES
Revised: 4. T RAINING AND INFERENCE PROCEDURES

Original: The training procedure is summarized in Algorithm 1.
Revised: The training procedure is summarized in Algorithm 1.

Original: Given random z ( i ) ∼ p ( z ) , we ﬁrst generate samples ˜ x
Revised: Given random z ( i ) ∼ p ( z ) , we ﬁrst generate samples ˜ x

Original: ( i ) ∼ p ( x | z
Revised: ( i ) ∼ p ( x | z

Original: ( i ) ) using the generator.
Revised: ( i ) ) using the generator.

Original: Then the discriminator is updated to distinguish between generated and real samples.
Revised: Then the discriminator is updated to distinguish between generated and real samples.

Original: The encoder outputs the parameters for the distribution q ( z | x ) , from which we calculate the log-likelihood in (II).
Revised: The encoder outputs the parameters for the distribution q ( z | x ) , from which we calculate the log-likelihood in (II).

Original: Then the generator and encoder are updated together to minimize the reconstruction error (i.e. maximize the expected log-likelihood), while the generator has an extra goal that is to fool the discriminator.
Revised: Then the generator and encoder are updated together to minimize the reconstruction error (i.e. maximize the expected log-likelihood), while the generator has an extra goal that is to fool the discriminator.

Original: The inference procedure is in Algorithm2.
Revised: The inference procedure is in Algorithm

Original: For any data x ( i ) , its inferred latent code is set to be the conditional mean µ ( x ( i ) )
Revised: For any data x ( i ) , its inferred latent code is set to be the conditional mean µ ( x ( i ) )

Original: =
Revised: =

Original: E q ( z | x ( i ) )
Revised: E q ( z | x ( i ) )

Original: [ z ] .
Revised: [ z ] .

Original: Then the reconstruction of x ( i ) is G ( µ ( x ( i ) )) .
Revised: Then the reconstruction of x ( i ) is G ( µ ( x ( i ) )) .

Original: Besides the reconstruction, we can also generate more samples which are close to x ( i ) in the sense that they have similar latent codes.
Revised: Besides the reconstruction, we can also generate more samples which are close to x ( i ) in the sense that they have similar latent codes.

Original: This can be done by ﬁrst sampling z ’s from the posterior q ( z | x ( i ) ) , and then map them to the data space using the generator.
Revised: This can be done by ﬁrst sampling z ’s from the posterior q ( z | x ( i ) ) , and then map them to the data space using the generator.

Original: [Table]
Revised: [Table]

Original: [Table]{+Algorithm 
 The AIM inference procedure.+}
Revised: {+Algorithm 
 The AIM inference procedure.+}

Original:  i ) ) ˜ x 
Revised: x ( i ) ∼ q ( x )

Revised: ( µ ( x ( i ) ) , σ 2 ( x ( i ) )) ← E ( x ( i ) )

Original:  ( i ) ← G ( µ ( x ( i ) ))
Revised: ˜ x ( i ) ← G ( µ ( x ( i ) ))

Original: (cid:46)

Original: Get reconstruction of x ( i ) z ( j ) ∼ N ( µ ( x ( i ) ) , σ 2 ( x ( i ) )) , j =1 , . . . , m (cid:46) Sample z from the posterior q ( z | x ( i ) )
Revised: z ( j ) ∼ N ( µ ( x ( i ) ) , σ 2 ( x ( i ) )) , j =

Original: ˜ x
Revised: ˜ x

Original: ( i,j ) ← G ( z ( j ) ) , j =1 , . . .
Revised: ( i,j ) ← G ( z ( j ) ) , j =

Original: , m (cid:46) Get similar neighbors of x ( i )

Original: 4. R EPARAMETRIZATION
Revised: 4. R EPARAMETRIZATION

Original: In order to sample from p ( x | z ) and q ( z | x ) , we use the reparametrization trick (Kingma and Welling, 2013; Bengio et al., 2013; 2014).
Revised: In order to sample from p ( x | z ) and q ( z | x ) , we use the reparametrization trick (Kingma and Welling, 2013; Bengio et al., 2013; 2014).

Original: Instead of directly sampling from a complex distribution, we can reparametrize the random variable as a deterministic transformation of an auxiliary noise variable (cid:15) , that is, u = f ( v, (cid:15) ) .
Revised: Instead of directly sampling from a complex distribution, we can reparametrize the random variable as a deterministic transformation of an auxiliary noise variable (cid:15) , that is, u = f ( v, (cid:15) ) .

Original: In our case, to sample from ˜ x ∼ p ( x | z ) , we can rewrite ˜ x
Revised: In our case, to sample from ˜ x ∼ p ( x | z ) , we can rewrite ˜ x

Original: = G ( z , (cid:15) ) , where z ∼ p ( z ) and (cid:15) is an auxiliary variable from a simple distribution.
Revised: = G ( z , (cid:15) ) , where z ∼ p ( z ) and (cid:15) is an auxiliary variable from a simple distribution.

Original: If we do not pass in any auxiliary noise (cid:15) , then the generator will be deterministic.
Revised: If we do not pass in any auxiliary noise (cid:15) , then the generator will be deterministic.

Original: For the encoder part, since q ( z | x ) = N ( µ ( x ) , σ 2 ( x ) I ) , one can draw samples by computing z = µ ( x ) + σ ( x ) (cid:12) (cid:15) , where (cid:15) ∼ N ( 0 , I ) .
Revised: For the encoder part, since q ( z | x ) = N ( µ ( x ) , σ 2 ( x ) I ) , one can draw samples by computing z = µ ( x ) + σ ( x ) (cid:12) (cid:15) , where (cid:15) ∼ N ( 0 , I ) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our proposed method, AIM, for both reconstruction and generation tasks, on the data sets MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky and Hinton, 2009) and CelebA (Liu et al., 2015).
Revised: We evaluate our proposed method, AIM, for both reconstruction and generation tasks, on the data sets MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky and Hinton, 2009) and CelebA (Liu et al., 2015).

Original: To show insights behind the effectiveness of AIM, we also conduct the same 2D Gaussian mixture experiment as in Dumoulin et al.
Revised: To show insights behind the effectiveness of AIM, we also conduct the same 2D Gaussian mixture experiment as in Dumoulin et al.

Original: (2016), and explore the 2D latent representation on MNIST.
Revised: (2016), and explore the 2D latent representation on MNIST.

Original: The architectures of our discriminator and generator are based on DCGAN (Radford et al., 2015) and slightly simpler, which can be easily replaced by more advanced state-of-the-art GANs, and we use a deterministic generator throughout the experiments.
Revised: The architectures of our discriminator and generator are based on DCGAN (Radford et al., 2015) and slightly simpler, which can be easily replaced by more advanced state-of-the-art GANs, and we use a deterministic generator throughout the experiments.

Original: Our encoder network consists of convolutional layers followed by two separated fully connected networks, which are used to predict the mean and variance of the posterior q ( z | x ) , respectively.
Revised: Our encoder network consists of convolutional layers followed by two separated fully connected networks, which are used to predict the mean and variance of the posterior q ( z | x ) , respectively.

Original: The Adam optimizer (Kingma and Ba, 2014) is used and the learning rate decay strategy suggested by Kingma and Ba (2014) is applied.
Revised: The Adam optimizer (Kingma and Ba, 2014) is used and the learning rate decay strategy suggested by Kingma and Ba (2014) is applied.

Original: Since the input to the log-function is one-dimensional in equation 

Revised: Since the input to the log-function is one-dimensional in equation

Original: but d -dimensional in equation 6, we set λ to be 1 /d .
Revised: but d -dimensional in equation 8, we set λ to be 1 /d .

Original: We also observe that the discriminator shares a similar task with the encoder: both of them need to extract higher level features from raw images.
Revised: We also observe that the discriminator shares a similar task with the encoder: both of them need to extract higher level features from raw images.

Original: Therefore, in order to reduce the number of parameters and to stabilize the training procedure, our encoder takes the intermediate hidden representation learned by the discriminator as its own input.
Revised: Therefore, in order to reduce the number of parameters and to stabilize the training procedure, our encoder takes the intermediate hidden representation learned by the discriminator as its own input.

Original: It is worth noting that the encoder does not update the common feature extracting layers.
Revised: It is worth noting that the encoder does not update the common feature extracting layers.

Original: More details about the model architecture and optimization methods are listed in the appendices.
Revised: More details about the model architecture and optimization methods are listed in the appendices.

Original: 5. Q UANTITATIVE RESULTS ON REAL DATASETS
Revised: 5. Q UANTITATIVE RESULTS ON REAL DATASETS

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we use quantitative measures to compare the inference and generation performance of
Revised: we use quantitative measures to compare the inference and generation performance of

Original: AIM, GAN, ALI and ALICE.
Revised: AIM, GAN, ALI and ALICE.

Original: And for fair comparison, GAN is implemented to have the identical generator and discriminator with AIM.
Revised: And for fair comparison, GAN is implemented to have the identical generator and discriminator with AIM.

Original: We also include a reduced version of AIM, named AIM − , in which the conditional distribution q ( z | x ) of the encoder is assumed to be a Gaussian with identity covariance matrix.
Revised: We also include a reduced version of AIM, named AIM − , in which the conditional distribution q ( z | x ) of the encoder is assumed to be a Gaussian with identity covariance matrix.

Original: To evaluate the performance of inference, we measure it through reconstructing test images and calculating the mean squared error (MSE), which has been adopted in Li et al. 
Revised: To evaluate the performance of inference, we measure it through reconstructing test images and calculating the mean squared error (MSE), which has been adopted in Li et al. [Table]

Original: As for generation, we calculate the inception score (Salimans et al., 2016) on 50 , 000 randomly generated images.
Revised: As for generation [Table] we calculate the inception score (Salimans et al., 2016) on 50 , 000 randomly generated images.

Original: The inception scores on MNIST are evaluated by the pre-trained classiﬁer from
Revised: The inception scores on MNIST are evaluated by the pre-trained classiﬁer from

Original: Li et al.
Revised: Li et al.

Original: [Table]
Revised: [Table]

Original: and the inception scores on CIFAR-10 is based on the ImageNet.
Revised: and the inception scores on CIFAR-10 is based on the ImageNet.

Original: The results are summarized in Table
Revised: The results are summarized in Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Inference From Table 1, AIM achieves the best reconstruction results on both data sets.
Revised: Inference From Table 1, AIM achieves the best reconstruction results on both data sets.

Original: On MNIST,
Revised: On MNIST,

Original: AIM signiﬁcantly decreases the MSE by 68% and 95% compared with ALICE and ALI respectively.
Revised: AIM signiﬁcantly decreases the MSE by 68% and 95% compared with ALICE and ALI respectively.

Original: On the more complicated CIFAT-10 data set, AIM decreases the MSE by 95% and 97%.
Revised: On the more complicated CIFAR-10 data set, AIM decreases the MSE by 95% and 97%.

Original: In order to alleviate the non-identiﬁable issue of ALI, ALICE adds the conditional entropy constraint by explicitly regularizing the l k norms between the reconstructed and real images.
Revised: In order to alleviate the non-identiﬁable issue of ALI, ALICE adds the conditional entropy constraint by explicitly regularizing the l k norms between the reconstructed and real images.

Original: However, as the data distribution becomes more complicated like in CIFAR-10, the l k norms become inadequate to measure the reconstruction.
Revised: However, as the data distribution becomes more complicated like in CIFAR-10, the l k norms become inadequate to measure the reconstruction.

Original: Consequently, ALICE’s reconstruction error on CIFAR-10 increases signiﬁcantly compared with that on MNIST.
Revised: Consequently, ALICE’s reconstruction error on CIFAR-10 increases signiﬁcantly compared with that on MNIST.

Original: In contrast, the reconstruction performance of AIM is consistent on both data sets.
Revised: In contrast, the reconstruction performance of AIM is consistent on both data sets.

Original: The reason is that our model explicitly speciﬁes the dependency structure of the generative model, and matches both prior and conditional distributions without using the simple data-ﬁtting l k metrics in the data space.
Revised: The reason is that our model explicitly speciﬁes the dependency structure of the generative model, and matches both prior and conditional distributions without using the simple data-ﬁtting l k metrics in the data space.

Original: This can be further justiﬁed by the performance of
Revised: This can be further justiﬁed by the performance of

Original: AIM − which follows the same structure.
Revised: AIM − which follows the same structure.

Original: Compared with AIM − , AIM further decreases the MSE signiﬁcantly by 49% on CIFAR-10, which shows that the inferred conditional variance is crucial for achieving the faithful reconstructions on complicated data sets.
Revised: Compared with AIM − , AIM further decreases the MSE signiﬁcantly by 49% on CIFAR-10, which shows that the inferred conditional variance is crucial for achieving the faithful reconstructions on complicated data sets.

Original: Generation AIM outperforms all the baseline models including GAN on inception score.
Revised: Generation AIM outperforms all the baseline models including GAN on inception score.

Original: This suggests that AIM can bring further improvement on generation performance instead of deteriorating it like the other baselines.
Revised: This suggests that AIM can bring further improvement on generation performance instead of deteriorating it like the other baselines.

Original: The reason that both ALI and ALICE perform worse than GAN on generation is that the task of matching two complicated joint distributions, p ( z , x ) and q ( z , x ) , is more difﬁcult than the task of the regular GAN, which is to match only the marginals, p ( x ) and q ( x ) .
Revised: The reason that both ALI and ALICE perform worse than GAN on generation is that the task of matching two complicated joint distributions, p ( z , x ) and q ( z , x ) , is more difﬁcult than the task of the regular GAN, which is to match only the marginals, p ( x ) and q ( x ) .

Original: The proposed model AIM explicitly deﬁnes the dependency structure between z and x , which is more effective compared with one step joint distribution matching.
Revised: The proposed model AIM explicitly deﬁnes the dependency structure between z and x , which is more effective compared with one step joint distribution matching.

Original: Comparison between AIM and AIM − shows that the learned variance is also critical for better generation performance.
Revised: Comparison between AIM and AIM − shows that the learned variance is also critical for better generation performance.

Original: We also want to highlight that AIM’s generation performance can be further improved by replacing the adversarial network with more advanced state-of-the-art GANs.
Revised: We also want to highlight that AIM’s generation performance can be further improved by replacing the adversarial network with more advanced state-of-the-art GANs.

Original: 5. R ECONSTRUCTION RESULTS
Revised: 5. R ECONSTRUCTION RESULTS

Original: We compare the reconstruction produced by AIM with the results of joint distribution regimes.
Revised: We compare the reconstruction produced by AIM with the results of joint distribution regimes.

Original: Since
Revised: Since

Original: ALICE is sensitive to hyper-parameter, we show its reconstruction results with different hyperparameter settings in Appendix, from which we observe either unfaithful reconstructions or blurriness.
Revised: ALICE is sensitive to hyper-parameter, we show its reconstruction results with different hyperparameter settings in Appendix, from which we observe either unfaithful reconstructions or blurriness.

Original: In Figure 2, we compare reconstruction of AIM with the results reported in ALI(Dumoulin et al., 2016)
Revised: In Figure [Table] we compare reconstruction of AIM with the results reported in ALI(Dumoulin et al., 2016)

Original: (BiGAN (Donahue et al., 2016)).
Revised: (BiGAN (Donahue et al., 2016)).

Original: From the ﬁrst column of Figure 2, we observe that ALI provides a certain level of reconstructions.
Revised: From the ﬁrst column of Figure 2, we observe that ALI provides a certain level of reconstructions.

Original: However, it fails to capture the precise style of the original digits.
Revised: However, it fails to capture the precise style of the original digits.

Original: In contrast, AIM can achieve very sharp and faithful reconstructions.
Revised: In contrast, AIM can achieve very sharp and faithful reconstructions.

Original: On the more complicated dataset CIFAR-10, ALI’s reconstructions are less faithful and oftentimes make mistakes in capturing exact object placement, color, style, and object identity.
Revised: On the more complicated dataset CIFAR-10, ALI’s reconstructions are less faithful and oftentimes make mistakes in capturing exact object placement, color, style [Table] and object identity.

Original: Our model produces better reconstructions in all these aspects.
Revised: Our model produces better reconstructions in all these aspects.

Original: For the reconstructions of CelebA in column three, AIM reproduces the similar style, color and face placement, and even achieves a high level of face identity.
Revised: For the reconstructions of CelebA in column three, AIM reproduces the similar style, color and face placement, and even achieves a high level of face identity.

Original: As stated in Dumoulin et al.
Revised: As stated in Dumoulin et al.

Original: , they believe ALI’s unfaithful reconstructions is caused by underﬁtting.
Revised: , they believe ALI’s unfaithful reconstructions is caused by underﬁtting.

Original: This also leads us to believe that our adversarial regime (marginal and conditional distribution matching) is more efﬁcient for inference compared to joint distribution matching regimes.
Revised: This also leads us to believe that our adversarial regime (marginal and conditional distribution matching) is more efﬁcient for inference compared to joint distribution matching regimes.

Original: 5. U NSUPERVISED CONDITIONAL GENERATION
Revised: 5. M ODE COLLAPSE REDUCTION

Original: Because the explicit posterior distribution q ( z | x ) can be learned by the proposed model AIM, we can generate the similar samples conditioned on a given sample.

Original: This is meaningful in downstream task like data augmentation. In Figure 3, based on any image x ( i ) in the ﬁrst column, we conditionally generate more images by sampling z from the posterior q ( z | x ( i ) ) . The conditionally generated images are shown in the same row of x ( i ) . We observe that they have the similar style, color, and face placement with the original image.
Revised: [Table]

Original: 5. M ODE COLLAPSE REDUCTION

Original: To show the effectiveness of our model on mode collapse reduction, we perform the same synthetic experiment as in Dumoulin et al.
Revised: To show the effectiveness of our model on mode collapse reduction, we perform the same synthetic experiment as in Dumoulin et al.

Original: The data is a 2D Gaussian mixture of 25 components laid out on a grid as shown in Figure 4a.
Revised: The data is a 2D Gaussian mixture of 25 components laid out on a grid as shown in Figure 3a.

Original: As reported in Dumoulin et al.
Revised: To quantify the degree of mode collapse, we use the two metrics used in Srivastava et al.

Original: (2016), on average, ALI covered 13.4 ± 5.8 modes (min: 8, max: 25), while GAN covered 10.
Revised: (2017): the number of modes captured and the percentage of high quality samples .

Original: ± 9.2 (min: 1, max: 22).
Revised: A generated sample is counted as high quality if it is within three standard deviations of the nearest mode. Then the number of modes captured is the number of mixture components whose mean is nearest to at least one high quality sample. As shown in Table 2, the proposed model AIM outperforms the other state-of-the-art models consistently on both measures.

Original: Under the same setting, we repeat the experiment for 10 times and our model can capture 25 modes every time.
Revised: More speciﬁcally, AIM can capture 25 modes every time and generate more than 80% of high-quality samples.

Original: This indicates that  AIMcan signiﬁcantly help overcome thechallenge of mode collapse.
Revised: This suggests that the proposed model AIM signiﬁcantly alleviates the mode collapse issue of the GAN framework and hence further improves the generation performance.

Original: [Figure] The inferred latent codes, reconstructions and generated samples by both models are reported in 
Revised: To further demonstrate the insights behind our model, we show the inferred latent codes, reconstructions, and generated samples by AIM together with the results by ALI in Figure 3.

Original: Figure 4.
Revised: Note that the ALI’s result here is produced by the best-covering model in their paper, while the average number of modes captured by ALI is 15.84 compared to ours 25.

Original: In Figure 4b, we observe that our latent codes exhibit a disk shape and lie in the three standard deviation area of the standard Gaussian prior.
Revised: From Figure 3b, we observe that our latent codes exhibit a disk shape and lie within the three standard deviations area of the standard Gaussian prior.

Original: They also keep the relative spacial relationship of the test samples.
Revised: Furthermore, they keep the relative spacial relationship of the test samples.

Original: From Figure 4b, we observe that AIM’s reconstruction is almost identical to the original data (Figure 4a), while ALI’s reconstruction is less faithful and consists of connecting points between modes.
Revised: In Figure 3c, AIM’s reconstruction is almost identical to the original data (Figure 3a), while ALI’s reconstruction is less faithful and consists of connecting points between modes.

Original: Both models have similar generating performance.
Revised: Both models have similar generating performance as shown in Figure 3d.

Original: Some generated samples lie in the regions between the true modes.
Revised: Some generated samples lie in the regions between the true modes.

Original: This is because the true modes are separated by large low-probability regions, which does not align well with the assumption of continuous prior distribution. 
Revised: This is because the true modes are separated by large low-probability regions. It does not align well with the assumption of continuous prior distribution. [Figure]

Original: 5. L ATENT REPRESENTATION OF MNIST
Revised: 5. L ATENT REPRESENTATION OF MNIST

Original: [Figure]
Revised: [Figure]

Original: The 2D latent codes of MNIST test data inferred by AIM is shown in Figure 10a.
Revised: The 2D latent codes of MNIST test data inferred by AIM is shown in Figure 10a.

Original: In a totally unsupervised manner, AIM does a great job at clustering the similar digits.
Revised: In a totally unsupervised manner, AIM does a great job at clustering the similar digits.

Original: The latent codes still lie in the standard Gaussian range but now exhibit no "holes", compared to Figure 4b.
Revised: The latent codes still lie in the standard Gaussian range but now exhibit no "holes", compared to Figure 3b.

Original: This is because the data distribution of MNIST is much more continuous than the Gaussian mixture.
Revised: This is because the data distribution of MNIST is much more continuous than the Gaussian mixture.

Original: In Figure 10c, we show images generated by linearly interpolating between − 2 and 2 along each dimension of the latent space.
Revised: In Figure 10c, we show images generated by linearly interpolating between − 2 and 2 along each dimension of the latent space.

Original: The generated images are sharp and the transitions between digits are smooth.
Revised: The generated images are sharp and the transitions between digits are smooth.

Original: This indicates that AIM learns smooth and meaningful representations which can generalize well.
Revised: This indicates that AIM learns smooth and meaningful representations which can generalize well.

Revised: 5. U NSUPERVISED CONDITIONAL GENERATION

Revised: Because the explicit posterior distribution q ( z | x ) can be learned by the proposed model AIM, we can generate the similar samples conditioned on a given sample.

Revised: This is meaningful in downstream task like data augmentation.

Original:   C ONCLUSION
Revised: In Figure 5, based on any image x ( i ) in the ﬁrst column, we conditionally generate more images by sampling z from the posterior q ( z | x ( i ) ) . The conditionally generated images are shown in the same row of x ( i ) . We observe that they have the similar style, color, and face placement with the original image.  C ONCLUSION

Original: We proposed a novel framework, AIM, which matches both prior and conditional distributions between the generator and the encoder.
Revised: We proposed a novel framework, AIM, which matches both prior and conditional distributions between the generator and the encoder.

Original: Adversarial inference is incorporated into this framework and there is no parametric assumption on the conditional data distribution.
Revised: Adversarial inference is incorporated into this framework and there is no parametric assumption on the conditional data distribution.

Original: Therefore, the proposed approach can not only learn an efﬁcient inference mechanism but also improve the ﬁdelity of generated samples.
Revised: Therefore, the proposed approach can not only learn an efﬁcient inference mechanism but also improve the ﬁdelity of generated samples.

Original: Extensive experiments on real datasets validate the effectiveness of the proposed model, and some insights are revealed by experiments on synthetic data.
Revised: Extensive experiments on real datasets validate the effectiveness of the proposed model, and some insights are revealed by experiments on synthetic data.

Original: A M ORE E XPERIMENT R ESULTS A. C OMPARISON BETWEEN AIM’ S AND ALICE’ S RECONSRUCTIONS ON C ELEB A
Revised: A M ORE E XPERIMENT R ESULTS A. C OMPARISON BETWEEN AIM’ S AND ALICE’ S RECONSRUCTIONS ON C ELEB A

Original: ALICE uses hyper-parameter λ to control the strength of the conditional entropy (CE) regularization in the objective function.
Revised: ALICE uses hyper-parameter λ to control the strength of the conditional entropy (CE) regularization in the objective function.

Original: When λ is 0, ALICE’s objective function reduces to ALI’s.
Revised: When λ is 0, ALICE’s objective function reduces to ALI’s.

Original: As observed from Fig 7, ALICE fails to reconstruct original images when hyper-parameter λ is relative small.
Revised: As observed from Fig 7, ALICE fails to reconstruct original images when hyper-parameter λ is relative small.

Original: This is related to the non-identiﬁability issue of ALI as described in Li et al.
Revised: This is related to the non-identiﬁability issue of ALI as described in Li et al.

Original: As λ increases, more patterns are captured, but the reconstructed images becomes more like the results of VAE – the images are all blurred.
Revised: As λ increases, more patterns are captured, but the reconstructed images becomes more like the results of VAE – the images are all blurred.

Original: This is because ALICE replaces the intractable CE with its upper bound cycle-consistency loss, which becomes l k norms under parametric assumption on p ( x | z ) .
Revised: This is because ALICE replaces the intractable CE with its upper bound cycle-consistency loss, which becomes l k norms under parametric assumption on p ( x | z ) .

Original: So just like VAE-GAN hybrids, ALICE can be treated as a VAE-ALI hybrids and also suffer from similar drawbacks.
Revised: So just like VAE-GAN hybrids, ALICE can be treated as a VAE-ALI hybrids and also suffer from similar drawbacks.

Original: Therefore, adding conditional entropy into ALI’s objective function heuristically manifests a compromise of the strengths and weaknesses of ALI and VAE.
Revised: Therefore, adding conditional entropy into ALI’s objective function heuristically manifests a compromise of the strengths and weaknesses of ALI and VAE.

Original: In contrast, Figure 6 shows that our proposed model AIM can overcome the limitations of ALI and
Revised: In contrast, Figure 6 shows that our proposed model AIM can overcome the limitations of ALI and

Original: VAE.
Revised: VAE.

Original: AIM’s reconstructions capture the identiﬁable features in the original images, and are also very sharp without blurriness.
Revised: AIM’s reconstructions capture the identiﬁable features in the original images, and are also very sharp without blurriness.

Original: In fact, we observe that AIM is able to reconstruct the images’ background (color, texture), which are always lost in ALICE’s reconstructions for any choice of λ .
Revised: In fact, we observe that AIM is able to reconstruct the images’ background (color, texture), which are always lost in ALICE’s reconstructions for any choice of λ .

Original: A. A NALYSIS OF R ECONSTRUCTION E RRORS
Revised: A. A NALYSIS OF R ECONSTRUCTION E RRORS

Original: We present the change of mean squared errors (MSE) on the ﬁrst 50 training epochs in Figure 8.
Revised: We present the change of mean squared errors (MSE) on the ﬁrst 50 training epochs in Figure 8.

Original: As can be seen, on all datasets, the MSEs decrease signiﬁcantly during the ﬁrst 10 epochs and then stay steadily at the similar minimum levels.
Revised: As can be seen, on all datasets, the MSEs decrease signiﬁcantly during the ﬁrst 10 epochs and then stay steadily at the similar minimum levels.

Original: Although the scales of reconstruction errors on different datasets at the beginning are different because of the various complexity of data distributions, the converged values of reconstruction errors are very similar.
Revised: Although the scales of reconstruction errors on different datasets at the beginning are different because of the various complexity of data distributions, the converged values of reconstruction errors are very similar.

Original: This observation proves that our inference mechanism can efﬁciently and consistently infer the latent codes of data samples, even when the data distribution is complicated.
Revised: This observation proves that our inference mechanism can efﬁciently and consistently infer the latent codes of data samples, even when the data distribution is complicated.

Original: [Figure]
Revised: [Figure]

Original: A. U NSUPERVISED C ONDITIONALLY G ENERATED I MAGES
Revised: A. U NSUPERVISED C ONDITIONALLY G ENERATED I MAGES

Original: In Figure 9, we present (unsupervised) conditionally generated samples of MNIST and CelebA experiments.
Revised: In Figure 9, we present (unsupervised) conditionally generated samples of MNIST and CelebA experiments.

Original: Each element in the ﬁrst column stands for an original data x ( i ) .
Revised: Each element in the ﬁrst column stands for an original data x ( i ) .

Original: Then based on x ( i ) , we conditionally generate more images by sampling z from the posterior q ( z | x ( i ) ) .
Revised: Then based on x ( i ) , we conditionally generate more images by sampling z from the posterior q ( z | x ( i ) ) .

Original: The conditionally generated images are shown in the same row of x ( i ) .
Revised: The conditionally generated images are shown in the same row of x ( i ) .

Original: A. R ANDOMLY G ENERATED I MAGES
Revised: A. R ANDOMLY G ENERATED I MAGES

Original: In Figure 10, we show randomly generated images by sampling z from the marginal p ( z ) .
Revised: In Figure 10, we show randomly generated images by sampling z from the marginal p ( z ) .

Original: The generation results can be easily improved by replacing DCGAN structure with more advanced GAN structures.
Revised: The generation results can be easily improved by replacing DCGAN structure with more advanced GAN structures.

Original: B H YPERPARAMETERS FOR MNIST
Revised: B H YPERPARAMETERS FOR MNIST

Original: The details for the networks used in MNIST experiment are presented in Table 2.
Revised: The details for the networks used in MNIST experiment are presented in Table 3.

Original: Here the column "feature maps" for the fully-connected (FC) layers represents the dimension of their output.
Revised: Here the column "feature maps" for the fully-connected (FC) layers represents the dimension of their output.

Original: FC ( µ )
Revised: FC ( µ )

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: and FC ( σ ) are two fully-connected layers built on the previous common layer to separately predict mean µ ( x ) and variance σ 2 ( x ) of the conditional distribution q ( z | x ) .
Revised: and FC ( σ ) are two fully-connected layers built on the previous common layer to separately predict mean µ ( x ) and variance σ 2 ( x ) of the conditional distribution q ( z | x ) .

Original: C H YPERPARAMETERS FOR CIFAR-10 AND C ELEB A
Revised: C H YPERPARAMETERS FOR CIFAR-10 AND C ELEB A

Original: The details for the networks used in CIFAR-10 and CelebA experiments are presented in Table 3.
Revised: The details for the networks used in CIFAR-10 and CelebA experiments are presented in Table 4.

Original: In order to reuse the same architecture for both datasets, we scaled CIFAR-10 images and center cropped CelebA images so that they have the same image size 64 × 64.
Revised: In order to reuse the same architecture for both datasets, we scaled CIFAR-10 images and center cropped CelebA images so that they have the same image size 64 × 64.

Original: [Table] [Table]
Revised: [Table] [Table]