Original: P AIRWISE A UGMENTED GAN S WITH A DVERSARIAL R ECONSTRUCTION L OSS
Revised: P AIRWISE A UGMENTED GAN S WITH A DVERSARIAL R ECONSTRUCTION L OSS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a novel autoencoding model called Pairwise Augmented GANs.
Revised: We propose a novel autoencoding model called Pairwise Augmented GANs.

Original: We train a generator and an encoder jointly and in an adversarial manner.
Revised: We train a generator and an encoder jointly and in an adversarial manner.

Original: The generator network learns to sample realistic objects.
Revised: The generator network learns to sample realistic objects.

Original: In turn the encoder network at the same timein turn is trained to map the true data distribution to the prior ina latent space.
Revised: In turn, the encoder network at the same time is trained to map the true data distribution to the prior in latent space.

Original: To ensure good reconstructions we introduce an augmented adversarial reconstruction loss.
Revised: To ensure good reconstructions, we introduce an augmented adversarial reconstruction loss.

Original: Here we train a discriminator to distinguish two types of pairs: the object with its augmentation and the one with its reconstruction.
Revised: Here we train a discriminator to distinguish two types of pairs: an object with its augmentation and the one with its reconstruction.

Original: We show that such adversarial loss compares objects based on the content rather than on the exact match.
Revised: We show that such adversarial loss compares objects based on the content rather than on the exact match.

Original: We experimentally demonstrate that our model generates samples and reconstructions of quality competitive with state-of-the-art on datasets MNIST, CIFAR10, CelebA and achieves good quantitative results on CIFAR
Revised: We experimentally demonstrate that our model generates samples and reconstructions of quality competitive with state-of-the-art on datasets MNIST, CIFAR10, CelebA and achieves good quantitative results on CIFAR

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep generative models are  powerful tool to sample complex high dimensional objects from a low dimensional manifold.
Revised: Deep generative models are a powerful tool to sample complex high dimensional objects from a low dimensional manifold.

Original: The dominant approaches for learning such generative models are variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014).
Revised: The dominant approaches for learning such generative models are variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014).

Original: VAEs allow not only to generate samples from the data distribution, but also to encode the objects into the latent space.
Revised: VAEs allow not only to generate samples from the data distribution, but also to encode the objects into the latent space.

Original: However, VAE-like models require a careful likelihood choice.
Revised: However, VAE-like models require a careful likelihood choice.

Original: Misspecifying one, may lead to undesirable effects in samples and reconstructions (e.g. blurry images).
Revised: Misspecifying one may lead to undesirable effects in samples and reconstructions (e.g., blurry images).

Original: On the contrary, GANs do not rely on an explicit likelihood and utilize more complex loss function provided by a discriminator.
Revised: On the contrary, GANs do not rely on an explicit likelihood and utilize more complex loss function provided by a discriminator.

Original: As a result, they produce higher quality images.
Revised: As a result, they produce higher quality images.

Original: However, the original formulation of GANs (Goodfellow et al., 2014) lacks an important encoding property that allows many practical applications.
Revised: However, the original formulation of GANs (Goodfellow et al., 2014) lacks an important encoding property that allows many practical applications.

Original: For example, it is used in a semisupervized learning (Kingma et al., 2014), in a manipulation of object properties using low dimensional manifold (Creswell et al., 2017) and in an optimization utilizing the known structure of embeddings (Gómez-Bombarelli et al., 2018).
Revised: For example, it is used in semi-supervised learning (Kingma et al., 2014), in a manipulation of object properties using low dimensional manifold (Creswell et al., 2017) and in an optimization utilizing the known structure of embeddings (Gómez-Bombarelli et al., 2018).

Original: VAE-GAN hybrids are of great interest due to their potential ability to learn latent representations like VAEs, while generating high quality objects like GANs.
Revised: VAE-GAN hybrids are of great interest due to their potential ability to learn latent representations like VAEs, while generating high-quality objects like GANs.

Original: In such generative models with a bidirectional mapping between the data space and the latent space one of  desired properties is to have good reconstructions ( x ≈ G ( E ( x )) ).
Revised: In such generative models with a bidirectional mapping between the data space and the latent space one of the desired properties is to have good reconstructions ( x ≈ G ( E ( x )) ).

Original: In many hybrid approaches (Rosca et al., 2017; Ulyanov et al., 2018; Zhu et al., 2017; Brock et al., 2017; Tolstikhin et al., 2017) as well as in VAE-like methods it is achieved by minimizing L 1 or L 2 pixel-wise norm between x and G ( E ( x )) .
Revised: In many hybrid approaches (Rosca et al., 2017; Ulyanov et al., 2018; Zhu et al., 2017; Brock et al., 2017; Tolstikhin et al., 2017) as well as in VAE-like methods it is achieved by minimizing L 1 or L 2 pixel-wise norm between x and G ( E ( x )) .

Original: However, the main drawback of using these standard reconstruction losses is that they enforce the generative model to recover too many redundant details of the source object x .
Revised: However, the main drawback of using these standard reconstruction losses is that they enforce the generative model to recover too many unnecessary details of the source object x .

Original: For example, to reconstruct a bird picture we do not need an exact position of the bird on an image, but the pixel-wise loss penalizes a lot for shifted reconstructions.
Revised: For example, to reconstruct a bird picture we do not need an exact position of the bird on an image, but the pixel-wise loss penalizes a lot for shifted reconstructions.

Original: Recently, Li et al.
Revised: Recently, Li et al.

Original: (2017) improved ALI model (Dumoulin et al., 2017; Donahue et al., 2017) by introducing a reconstruction loss in the form of a discriminator which classiﬁes pairs ( x, x ) and ( x, G ( E ( x ))) .
Revised: (2017) improved ALI model (Dumoulin et al., 2017; Donahue et al., 2017) by introducing a reconstruction loss in the form of a discriminator which classiﬁes pairs ( x, x ) and ( x, G ( E ( x ))) .

Original: However, in such approach the discriminator tends to detect the fake pair ( x, G ( E ( x ))) just by checking the identity of x and G ( E ( x )) which leads to vanishing gradients.
Revised: However, in such approach, the discriminator tends to detect the fake pair ( x, G ( E ( x ))) just by checking the identity of x and G ( E ( x )) which leads to vanishing gradients.

Original: In this paper, we propose a novel autoencoding model which matches the distributions in the data space and in the latent space independently as in Zhu et al.
Revised: In this paper, we propose a novel autoencoding model which matches the distributions in the data space and in the latent space independently as in Zhu et al.

Original: To ensure good reconstructions we introduce an augmented adversarial reconstruction loss as a discriminator which classiﬁes pairs ( x, a ( x )) and ( x, G ( E ( x ))) where a ( · ) is a stochastic augmentation function.
Revised: To ensure good reconstructions, we introduce an augmented adversarial reconstruction loss as a discriminator which classiﬁes pairs ( x, a ( x )) and ( x, G ( E ( x ))) where a ( · ) is a stochastic augmentation function.

Original: This enforces the x ∼ p ∗ ( x ) y ∼ r ( y | x )
Revised: This enforces the x ∼ p ∗ ( x ) y ∼ r ( y | x )

Original: D ( x , y )
Revised: D ( x , y )

Original: ˆ y ∼ p θ ( y | z )
Revised: ˆ y ∼ p θ ( y | z )

Original: z ∼ q ϕ ( z | x ) x ∼ p ∗ ( x ) a ( x )
Revised: z ∼ q ϕ ( z | x ) x ∼ p ∗ ( x ) a ( x )

Original: G ( z )
Revised: G ( z )

Original: E ( x ) ( x , y ) ( x , ˆ y )
Revised: E ( x ) ( x , y ) ( x , ˆ y )

Original: D ( z ) ˆ z ∼ q ϕ
Revised: D ( z ) ˆ z ∼ q ϕ

Original: ( z | x ) x ∼ p ∗ ( x ) z ∼ p ( z ) E ( x ) ˆ z z
Revised: ( z | x ) x ∼ p ∗ ( x ) z ∼ p ( z ) E ( x ) ˆ z z

Original: D ( x ) x ∼ p ∗ ( x ) ˆ x∼ p θ ( x | z )
Revised: D ( x ) x ∼ p ∗ ( x ) [Figure] x

Original:  z ∼ p
Revised: [Figure] ( x | z ) z ∼ p

Original: ( z )
Revised: ( z )

Original: G ( z ) x ˆ x discriminator to take into account content invariant to the augmentation, thus making training more robust.
Revised: G [Figure] [Figure] discriminator to take into account content invariant to the augmentation, thus making training more robust.

Original: We call this approach Pairwise Augmented Generative Adversarial Networks (PAGANs).
Revised: We call this approach Pairwise Augmented Generative Adversarial Networks (PAGANs).

Original: Measuring a reconstruction quality of autoencoding models is challenging.
Revised: Measuring a reconstruction quality of autoencoding models is challenging.

Original: A standard reconstruction metric RMSE does not perform  content-based comparison.
Revised: A standard reconstruction metric RMSE does not perform the content-based comparison.

Original: To deal with this problem we propose a novel metric Reconstruction Inception Dissimilarity (RID) which is robust to content-preserving transformations (e.g. small shifts of an image).
Revised: To deal with this problem we propose a novel metric Reconstruction Inception Dissimilarity (RID) which is robust to content-preserving transformations (e.g., small shifts of an image).

Original: We show qualitative results on common datasets such as MNIST (LeCun & Cortes, 2010), CIFAR10 (Krizhevsky et al., 2009) and CelebA
Revised: We show qualitative results on common datasets such as MNIST (LeCun & Cortes, 2010), CIFAR10 (Krizhevsky et al., 2009) and CelebA

Original: (Liu et al., 2015).
Revised: (Liu et al., 2015).

Original: PAGANs outputperform existing VAE-GAN hybrids in Inception Score (Salimans et al., 2016) and Fréchet Inception Distance (Heusel et al., 2017) except  the recently announced method PD-WGAN (Gemici et al., 2018) on CIFAR10 dataset.
Revised: PAGANs outperform existing VAE-GAN hybrids in Inception Score (Salimans et al., 2016) and Fréchet Inception Distance (Heusel et al., 2017) except for the recently announced method PD-WGAN (Gemici et al., 2018) on CIFAR10 dataset.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Let us considerthe framework of an adversarial learning  where our goal is to match the true distribution p ∗ ( x ) to the model distribution p θ ( x ) .
Revised: Let us consider an adversarial learning framework where our goal is to match the true distribution p ∗ ( x ) to the model distribution p θ ( x ) .

Original: As it was proposed in the original paper Goodfellow et al.
Revised: As it was proposed in the original paper Goodfellow et al.

Original: (2014), the model distribution p θ ( x ) is induced by the generator G θ : z −→ x where z is sampled from a prior p ( z ) .
Revised: (2014), the model distribution p θ ( x ) is induced by the generator G θ : z −→ x where z is sampled from a prior p ( z ) .

Original: In order to match the distributions p ∗ ( x ) and p θ ( x ) in an adversarial manner the discriminator D ψ : x −→ [0 , 1] was introduced.
Revised: To match the distributions p ∗ ( x ) and p θ ( x ) in an adversarial manner, we introduce a discriminator D ψ : x −→ [0 , 1] .

Original: It takes an object x and predicts the probability that this object is sampled from the true distribution p ∗ ( x ) .
Revised: It takes an object x and predicts the probability that this object is sampled from the true distribution p ∗ ( x ) .

Original: The training procedure of GANs (Goodfellow et al., 2014) is based on the minimax game of two players: the generator G θ and the disciminator D ψ .
Revised: The training procedure of GANs (Goodfellow et al., 2014) is based on the minimax game of two players: the generator G θ and the discriminator D ψ .

Original: This game is deﬁned as follows
Revised: This game is deﬁned as follows

Original: [Equation]
Revised: [Equation]

Original: where V ( θ, ψ ) is a value function for this game.
Revised: where V ( θ, ψ ) is a value function for this game.

Original: The optimal discriminator D ψ ∗ given ﬁxed generator G θ is
Revised: The optimal discriminator D ψ ∗ given ﬁxed generator G θ is

Original: [Equation]
Revised: [Equation]

Original: and then the value function for the generator V ( θ, ψ ∗ ) given the optimal discriminator D ψ ∗ is equivalent to the Jensen-Shanon divergence between the model distribution p θ ( x ) and the true distribution p ∗ ( x ) , i.e.
Revised: and then the value function for the generator V ( θ, ψ ∗ ) given the optimal discriminator D ψ ∗ is equivalent to the Jensen-Shanon divergence between the model distribution p θ ( x ) and the true distribution p ∗ ( x ) , i.e.

Original: [Equation]
Revised: [Equation]

Original: However, in practice the gradient of the value function V ( θ, ψ ) with respect to  generator’s parameters θ vanishes to zero.
Revised: However, in practice, the gradient of the value function V ( θ, ψ ) with respect to the generator’s parameters θ vanishes to zero.

Original: Therefore, Goodfellow et al.
Revised: Therefore, Goodfellow et al.

Original: (2014) proposed to train the generator G θ by minimizing − log D ψ ( G θ ( z )) instead of log(1 − D ψ ( G θ ( z ))) .
Revised: (2014) proposed to train the generator G θ by minimizing − log D ψ ( G θ ( z )) instead of log(1 − D ψ ( G θ ( z ))) .

Original: This loss for the generator provides much more stable gradients and has the same ﬁxed point as the minimax game of D ψ and G θ .
Revised: This loss for the generator provides much more stable gradients and has the same ﬁxed point as the minimax game of D ψ and G θ .

Original:  P AIRWISE A UGMENTED G ENERATIVE A DVERSARIAL N ETWORKS
Revised:  P AIRWISE A UGMENTED G ENERATIVE A DVERSARIAL N ETWORKS

Original: In PAGANs model our aim is not only to learn how to generate real objects with the generator
Revised: In PAGANs model our aim is not only to learn how to generate real objects with the generator

Original: G θ ( z ) where z is sampled from prior p ( z ), but at the same time learn an inverse mapping (encoder)
Revised: G θ ( z ) where z is sampled from prior p ( z ) but at the same time learn an inverse mapping (encoder)

Original: E ϕ : x −→ z .
Revised: E ϕ : x −→ z .

Original: Additionally, we use the third stochastic transformation a : x −→ y without parameters which is called augmenter.
Revised: Additionally, we use the third stochastic transformation a : x −→ y without parameters which is called augmenter.

Original: It produces the augmentation y of the source object x .
Revised: It produces the augmentation y of the source object x .

Original: Let us consider the distributions which are induced by these three mappings
Revised: Let us consider the distributions which are induced by these three mappings

Original: • p θ ( x | z ) - the conditional distribution of outputs of the generator G θ ( z ) given z ; • q ϕ ( z | x ) - the conditional distribution of outputs of the encoder E ϕ ( x ) given x ; • r ( y | x ) - the conditional distribution over the augmentations a ( x ) given a source object x .
Revised: • p θ ( x | z ) - the conditional distribution of outputs of the generator G θ ( z ) given z ; • q ϕ ( z | x ) - the conditional distribution of outputs of the encoder E ϕ ( x ) given x ; • r ( y | x ) - the conditional distribution over the augmentations a ( x ) given a source object x .

Original: Within the PAGANs model our goal is to ﬁnd such optimal parameters θ ∗ and ϕ ∗ that ensure
Revised: Within the PAGANs model our goal is to ﬁnd such optimal parameters θ ∗ and ϕ ∗ that ensure

Original: generator matching : p θ ∗ ( x ) =
Revised: generator matching : p θ ∗ ( x ) =

Original: p ∗ ( x ) where p θ ∗ ( x ) =
Revised: p ∗ ( x ) where p θ ∗ ( x ) =

Original: (cid:82) p θ ∗ ( x | z ) p ( z ) dz ,
Revised: (cid:82) p θ ∗ ( x | z ) p ( z ) dz ,

Original: i
Revised: i

Original: e. the generator G θ ∗ samples objects from the true distribution p ∗ ( x ) ; 2.
Revised: e. the generator G θ ∗ samples objects from the true distribution p ∗ ( x ) ; 2.

Original: encoder matching : q ϕ ∗ ( z ) =
Revised: encoder matching : q ϕ ∗ ( z ) =

Original: p ( z ) where q ϕ ∗ ( z ) =
Revised: p ( z ) where q ϕ ∗ ( z ) =

Original: (cid:82) q ϕ ∗ ( z | x ) p ∗ ( x ) dx , i.e. the encoder E ϕ generates embeddings z as the prior p ( z ) ; 3.
Revised: (cid:82) q ϕ ∗ ( z | x ) p ∗ ( x ) dx , i.e. the encoder E ϕ generates embeddings z as the prior p ( z ) ; 3.

Original: reconstruction matching : p θ∗ ,
Revised: reconstruction matching : p θ

Original: ϕ ∗ ( y | x
Revised: ∗ ,ϕ ∗ y | x

Original: =
Revised: =

Original: r
Revised: r

Original: ( y | x  where p
Revised: y | x ) where p

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: e. reconstructions G θ ∗ ( E ϕ ∗ ( x )) are distributed as augmentations r ( y | x ) of the source object x 
Revised: e. reconstructions G θ ∗ ( E ϕ ∗ ( x )) are distributed as augmentations r ( y | x ) of the source object x .

Original: 3. G ENERATOR & E NCODER M ATCHING
Revised: 3. G ENERATOR & E NCODER M ATCHING

Original: In order to deal with generator and encoder matching problems we can use the framework of the vanilla GANs (Goodfellow et al., 2014).
Revised: In order to deal with generator and encoder matching problems we can use the framework of the vanilla GANs (Goodfellow et al., 2014).

Original: We introduce two discriminators D ψ x and D ψ z for two minimax games:
Revised: We introduce two discriminators D ψ x and D ψ z for two minimax games:

Original: • generator matching:
Revised: • generator matching:

Original: [Equation]
Revised: [Equation]

Original: • encoder matching
Revised: • encoder matching

Original: [Equation]
Revised: [Equation]

Original: Then the value functions V x and V z given the optimal discriminators D ψ ∗ x and D ψ ∗ z are equivalent to
Revised: Then the value functions V x and V z given the optimal discriminators D ψ ∗ x and D ψ ∗ z are equivalent to

Original: Jensen-Shanon divergence:
Revised: Jensen-Shanon divergence:

Original: [Equation]
Revised: [Equation]

Original: 3. R ECONSTRUCTION M ATCHING : A UGMENTED A DVERSARIAL R ECONSTRUCTION L OSS
Revised: 3. R ECONSTRUCTION M ATCHING : A UGMENTED A DVERSARIAL R ECONSTRUCTION L OSS

Original: The solution of the reconstruction matching problem ensures that reconstructions G θ ( E ϕ ( x )) correspond to the source object x up to deﬁned random augmentations a ( x ) .
Revised: The solution of the reconstruction matching problem ensures that reconstructions G θ ( E ϕ ( x )) correspond to the source object x up to deﬁned random augmentations a ( x ) .

Original: In PAGANs model we introduce the minimax game for training the adversarial distance between the reconstructions and augmentations of the source object x .
Revised: In PAGANs model we introduce the minimax game for training the adversarial distance between the reconstructions and augmentations of the source object x .

Original: We consider the discriminator D ψ which takes a pair ( x, y ) and classiﬁes it into one of the following classes:
Revised: We consider the discriminator D ψ which takes a pair ( x, y ) and classiﬁes it into one of the following classes:

Original: • the real class: pairs ( x, y ) from the distribution p ∗ ( x ) r ( y | x ) , i.e. the object x is taken from the true distribution p ∗ ( x ) and the second y is obtained from the x by the random augmentation a ( x ) ; • the fake class: pairs ( x, y ) from the distribution
Revised: • the real class: pairs ( x, y ) from the distribution p ∗ ( x ) r ( y | x ) , i.e. the object x is taken from the true distribution p ∗ ( x ) and the second y is obtained from the x by the random augmentation a ( x ) ; • the fake class: pairs ( x, y ) from the distribution

Original: [Equation]
Revised: [Equation]

Original: i.e. x is sampled from p ∗ ( x ) then z is generated from the conditional distribution q ϕ
Revised: i.e. x is sampled from p ∗ ( x ) then z is generated from the conditional distribution q ϕ

Original: ( z | x ) by the encoder E ϕ ( x ) and y is produced by the generator G ϕ
Revised: ( z | x ) by the encoder E ϕ ( x ) and y is produced by the generator G ϕ

Original: ( z ) from the conditional model distribution p
Revised: ( z ) from the conditional model distribution p

Original: θ
Revised: θ

Original: ( y | z ) .
Revised: ( y | z ) .

Original: Then the minimax problem is
Revised: Then the minimax problem is

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Let us prove that such minimax game will match the distributions r ( y | x ) and p θ,ϕ ( y | x ) .
Revised: Let us prove that such minimax game will match the distributions r ( y | x ) and p θ,ϕ ( y | x ) .

Original: At ﬁrst, we ﬁnd the optimal discriminator:
Revised: At ﬁrst, we ﬁnd the optimal discriminator:

Original: Proposition 1.
Revised: Proposition 1.

Original: Given a ﬁxed generator G θ and a ﬁxed encoder E ϕ , the optimal discriminator D ψ ∗ is
Revised: Given a ﬁxed generator G θ and a ﬁxed encoder E ϕ , the optimal discriminator D ψ ∗ is

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Given in Appendix A.1.
Revised: Given in Appendix A.1.

Original: Then we can prove that given an optimal discriminator the value function V ( θ, ϕ, ψ ) is equivalent to the expected Jensen-Shanon divergence between the distributions r ( y | x ) and p θ,ϕ ( y | x ) .
Revised: Then we can prove that given an optimal discriminator the value function V ( θ, ϕ, ψ ) is equivalent to the expected Jensen-Shanon divergence between the distributions r ( y | x ) and p θ,ϕ ( y | x ) .

Original: Proposition 2.
Revised: Proposition 2.

Original: The minimization of the value function V under an optimal discriminator D ψ ∗ is equivalent to the minimization of the expected Jensen-Shanon divergence between r ( y | x ) and p θ,ϕ ( y | x ) , i.e.
Revised: The minimization of the value function V under an optimal discriminator D ψ ∗ is equivalent to the minimization of the expected Jensen-Shanon divergence between r ( y | x ) and p θ,ϕ ( y | x ) , i.e.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Given in Appendix A.2.
Revised: Given in Appendix A.2.

Original: If r ( y | x ) = δ x ( y ) then the optimal discriminator D ψ ∗ ( x, y ) will learn an indicator
Revised: If r ( y | x ) = δ x ( y ) then the optimal discriminator D ψ ∗ ( x, y ) will learn an indicator

Original: I { x = y } as was proved in Li et al.
Revised: I { x = y } as was proved in Li et al.

Original: As a consequence,  objectives of the generator and the encoder are very unstable and have vanishing gradients in practice.
Revised: As a consequence, the objectives of the generator and the encoder are very unstable and have vanishing gradients in practice.

Original: On the contrary, if the distribution r ( y | x ) is non-degenerate as in our model then the value function V ( θ, ϕ, ψ ) will be well-behaved and much more stable which we observed in practice.
Revised: On the contrary, if the distribution r ( y | x ) is non-degenerate as in our model then the value function V ( θ, ϕ, ψ ) will be well-behaved and much more stable which we observed in practice.

Original: 3. T RAINING O BJECTIVES
Revised: 3. T RAINING O BJECTIVES

Original: We obtain that for the generator and the encoder we should optimize the sum of two value functions:
Revised: We obtain that for the generator and the encoder we should optimize the sum of two value functions:

Original: • the generator’s objective:
Revised: • the generator’s objective:

Original: [Equation]
Revised: [Equation]

Original: • the encoder’s objective:
Revised: • the encoder’s objective:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: (cid:46)
Revised: (cid:46)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: , N
Revised: , N

Original: [Table]
Revised: [Table]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: , N
Revised: , N

Original: [Table]
Revised: [Table]

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: j
Revised: j

Original: [Table] [Table]
Revised: [Table] [Table]

Original: In practice in order to speed up the training we follow Goodfellow et al.
Revised: In practice in order to speed up the training we follow Goodfellow et al.

Original: and use more stable objectives replacing log(1 − D ( · )) with − log( D ( · )) .
Revised: and use more stable objectives replacing log(1 − D ( · )) with − log( D ( · )) .

Original: See Figure 1 for the description of our model and Algorithm 1 for an algorithmic illustration of the training procedure.
Revised: See Figure 1 for the description of our model and Algorithm 1 for an algorithmic illustration of the training procedure.

Original: We can straightforwardly extend the deﬁnition of PAGANs model to f -PAGANs which minimize the f -divergence and to WPAGANs which optimize the Wasserstein-1 distance.
Revised: We can straightforwardly extend the deﬁnition of PAGANs model to f -PAGANs which minimize the f -divergence and to WPAGANs which optimize the Wasserstein-1 distance.

Original: More detailed analysis of these models is placed in Appendix B.
Revised: More detailed analysis of these models is placed in Appendix C.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Recent papers on VAE-GAN hybrids explore different ways to build a generative model with an encoder part.
Revised: Recent papers on VAE-GAN hybrids explore different ways to build a generative model with an encoder part.

Original: One direction is to applyan adversarial training in the VAE frameworkin order to match the variational posterior distribution q ( z | x ) and the prior distribution p ( z ) (Mescheder et al., 2017) or to match the marginal q ( z ) and p ( z ) (Makhzani et al., 2016; Tolstikhin et al., 2017).
Revised: One direction is to apply adversarial training in the VAE framework to match the variational posterior distribution q ( z | x ) and the prior distribution p ( z ) (Mescheder et al., 2017) or to match the marginal q ( z ) and p ( z ) (Makhzani et al., 2016; Tolstikhin et al., 2017).

Original: Other way within  VAE model is to introduce the discriminator as a part of a data likelihood (Larsen et al., 2015; Brock et al., 2017).
Revised: Another way within the VAE model is to introduce the discriminator as a part of a data likelihood (Larsen et al., 2015; Brock et al., 2017).

Original: Within the GANs framework a common technique is to regularize the model with the reconstruction loss term (Che et al., 2017; Rosca et al., 2017; Ulyanov et al., 2018).
Revised: Within the GANs framework, a common technique is to regularize the model with the reconstruction loss term (Che et al., 2017; Rosca et al., 2017; Ulyanov et al., 2018).

Original: Another principal approach is to train the generator and the encoder (Donahue et al., 2017;
Revised: Another principal approach is to train the generator and the encoder (Donahue et al., 2017;

Original: Dumoulin et al., 2017; Li et al., 2017) simultaneously in a fully adversarial way.
Revised: Dumoulin et al., 2017; Li et al., 2017) simultaneously in a fully adversarial way.

Original: These methods match the joint distributions p ∗ ( x ) q ( z | x ) and p θ ( x | z ) p ( z ) by training the discriminator which classiﬁes the pairs ( x, z ) .
Revised: These methods match the joint distributions p ∗ ( x ) q ( z | x ) and p θ ( x | z ) p ( z ) by training the discriminator which classiﬁes the pairs ( x, z ) .

Original: ALICE model (Li et al., 2017) introduces an additional entropy loss for dealing with the non-identiﬁability issues in ALI model.
Revised: ALICE model (Li et al., 2017) introduces an additional entropy loss for dealing with the non-identiﬁability issues in ALI model.

Original: Li et al.
Revised: Li et al.

Original: (2017) approximated the entropy loss with the cycle-consistency term which is equivalent to the adversarial reconstruction loss.
Revised: (2017) approximated the entropy loss with the cycle-consistency term which is equivalent to the adversarial reconstruction loss.

Original: The model of Pu et al.
Revised: The model of Pu et al.

Original: (2017a) puts ALI to the VAE framework where the same joint distributions are matched in an adversarial manner.
Revised: (2017a) puts ALI to the VAE framework where the same joint distributions are matched in an adversarial manner.

Original: As an alternative, Ulyanov et al.
Revised: As an alternative, Ulyanov et al.

Original: (2018) train generator and encoder in an adversarial manner without the discriminator.
Revised: (2018) train generator and encoder by optimizing the minimax game without the discriminator.

Original: Optimal transport approach is also explored, Gemici et al.
Revised: Optimal transport approach is also explored, Gemici et al.

Original: (2018) introduce an algorithm based on primal and dual formulations of an optimal transport problem.
Revised: (2018) introduce an algorithm based on primal and dual formulations of an optimal transport problem.

Original: In PAGANs model the marginal distributions in the data space p ∗ ( x ) and p θ ( x ) and in the latent space p ( z ) and q ( z ) are matched independently as in Zhu et al.
Revised: In PAGANs model the marginal distributions in the data space p ∗ ( x ) and p θ ( x ) and in the latent space p ( z ) and q ( z ) are matched independently as in Zhu et al.

Original: Additionally, the augmented adversarial reconstruction loss is minimized by fooling the discriminator which classiﬁes the pairs ( x, a ( x )) and ( x, G ( E ( x ))) .
Revised: Additionally, the augmented adversarial reconstruction loss is minimized by fooling the discriminator which classiﬁes the pairs ( x, a ( x )) and ( x, G ( E ( x ))) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we validate our model experimentally.
Revised: In this section, we validate our model experimentally.

Original: At ﬁrst, we compare PAGAN with other similar methods that allow to perform both inference and generation using Inception Score and 
Revised: At ﬁrst, we compare PAGAN with other similar methods that allow performing both inference and generation using Inception Score and Fréchet

Original: Fréchet Inception Distance.
Revised: Inception Distance.

Original: Secondly, to measure reconstruction quality we introduce Reconstruction
Revised: Secondly, to measure reconstruction quality, we introduce Reconstruction

Original: Inception Dissimilarity (RID) and prove its usability.
Revised: Inception Dissimilarity (RID) and prove its usability.

Original: In  last two experiments we show an importance of the adversarial loss and augmentations.
Revised: In the last two experiments we show the importance of the adversarial loss and augmentations.

Original: For the architecture choice we used deterministic DCGAN 1 generator and discriminator networks provided by pfnet-research 2 , the encoder network has the same architecture as the discriminator except  the output dimension.
Revised: For the architecture choice we used deterministic DCGAN 1 generator and discriminator networks provided by pfnet-research 2 , the encoder network has the same architecture as the discriminator except for the output dimension.

Original: The encoder’s output is a factorized normal distribution.
Revised: The encoder’s output is a factorized normal distribution.

Original: Thus p θ ( x | z ) =
Revised: Thus p θ ( x | z ) =

Original: δ G θ ( z ) ( x ) , q ϕ ( z | x ) =
Revised: δ G θ ( z ) ( x ) , q ϕ ( z | x ) =

Original: N ( µ ϕ ( x ) , σ 2 ϕ ( x ) I ) where µ ϕ , σ ϕ are outputs of the encoder network.
Revised: N ( µ ϕ ( x ) , σ 2 ϕ ( x ) I ) where µ ϕ , σ ϕ are outputs of the encoder network.

Original: The discriminator D ( z ) architecture is chosen to be a 2 layer MLP with 512, 256 hidden units.
Revised: The discriminator D ( z ) architecture is chosen to be a 2 layer MLP with 512, 256 hidden units.

Original: We also used the same default hyperparameters as provided in the repository and applied a spectral normalization following Miyato et al.
Revised: We also used the same default hyperparameters as provided in the repository and applied a spectral normalization following Miyato et al.

Original: For the augmentation a ( x ) deﬁned in Section 3 we used a combination of reﬂecting 10% pad and the random crop to the same image size.
Revised: For the augmentation a ( x ) deﬁned in Sectionwe used a combination of reﬂecting 10% pad and the random crop to the same image size.

Original: The prior distribution p ( z ) is chosen to be a standard distribution N (0 , I ) .
Revised: The prior distribution p ( z ) is chosen to be a standard distribution N (0 , I ) .

Original: To evaluate Inception Score and
Revised: To evaluate Inception Score and

Original: Fréchet Inception Distance we used the ofﬁcial implementation provided in tensorflow 1.10.
Revised: Fréchet Inception Distance we used the ofﬁcial implementation provided in tensorflow 1.10.

Original: (Abadi et al., 2015).
Revised: (Abadi et al., 2015).

Original: To optimize objectives (16), (14) we need to have a discriminator working on pairs ( x, y ) .
Revised: To optimize objectives (16), (14), we need to have a discriminator working on pairs ( x, y ) .

Original: This can be done using special network architectures like siam networks (Bromley et al., 1993) or via an image concatenation.
Revised: This can be done using special network architectures like siam networks (Bromley et al., 1993) or via an image concatenation.

Original: The latter approach can be implemented in two concurrent ways: concatenating channel or width wise.
Revised: The latter approach can be implemented in two concurrent ways: concatenating channel or widthwise.

Original: Empirically we found that the siam architecture does not lead to signiﬁcant improvement and concatenating width wise to be the most stable.
Revised: Empirically we found that the siam architecture does not lead to signiﬁcant improvement and concatenating width wise to be the most stable.

Original: We use this conﬁguration in all the experiments.
Revised: We use this conﬁguration in all the experiments.

Original: Sampling Quality
Revised: Sampling Quality

Original: To see whether our method provides good quality samples from the prior
Revised: To see whether our method provides good quality samples from the prior

Original: [Table]
Revised: [Table]

Original: we compared our method to related works that allow an inverse mapping.
Revised: we compared our model to related works that allow an inverse mapping.

Original: We performed our evaluations on CIFAR10 dataset since quantitative metrics are available there.
Revised: We performed our evaluations on CIFAR10 dataset since quantitative metrics are available there.

Original: Considering Fréchet Inception Distance (FID), our model outperforms all other methods.
Revised: Considering Fréchet Inception Distance (FID), our model outperforms all other methods.

Original: Inception Score shows that PAGANs signiﬁcantly better than others except  recently announced PD-WGAN.
Revised: Inception Score shows that PAGANs signiﬁcantly better than others except for recently announced PD-WGAN.

Original: Quantitative results are given in Table 1. 
Revised: Quantitative results are given in Table 1. For

Revised: S-VAE we report IS that is reproduced using

Revised: ofﬁcially provided code and hyperparameters 3 .

Original: Plots with samples and reconstructions for CIFAR10 dataset are provided in Figure 2.
Revised: Plots with samples and reconstructions for CIFAR10 dataset are provided in Figure 2.

Original: Additional visual results for more datasets can be found in Appendix D.
Revised: Additional visual results for more datasets can be found in Appendix E.3.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Reconstruction Inception 
Revised: Reconstruction Inception Dissimilarity

Original: [Table]

Original: The traditional approach to estimate the reconstruction quality is to compute RMSE distance from source images to reconstructed ones.
Revised: The traditional approach to estimate the reconstruction quality is to compute RMSE distance from source images to reconstructed ones.

Original: However, this metric suffers from focusing on exact reconstruction and is not content aware.
Revised: However, this metric suffers from focusing on exact reconstruction and is not content aware.

Original: RMSE penalizes content preserving transformations while allowssuch undesirable effect as blurriness which degrades visual quality signiﬁcantly. We propose a novel metric Reconstruction Inception Dissimilarity (RID) which is based on a pretrained classiﬁcation
Revised: RMSE penalizes content-preserving transformations while allows

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  network and is deﬁned as follows:
Revised: Evaluation for SVAE is based on ofﬁcially provided code and hyperparameters. such undesirable effect as blurriness which degrades visual quality signiﬁcantly. We propose a novel metric Reconstruction Inception Dissimilarity (RID) which is based on a pre-trained classiﬁcation network and is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: 18) where p ( y | x ) is a pretrained classiﬁer that estimates the label distribution given an image.
Revised: (18) where p ( y | x ) is a pre-trained classiﬁer that estimates the label distribution given an image.

Original: Similar to Salimans et al.
Revised: Similar to Salimans et al.

Original: we use a pretrained Inception Network (Szegedy et al., 2016) to calculate softmax outputs.
Revised: we use a pre-trained Inception Network (Szegedy et al., 2016) to calculate softmax outputs.

Original: [Table]
Revised: [Table]

Original: Low RID indicates that the content did not change after reconstruction.
Revised: Low RID indicates that the content did not change after reconstruction.

Original: To calculate standard deviations we use  same approach as for IS and split test set on 10 equal parts 3 .
Revised: To calculate standard deviations, we use the same approach as for IS and split test set on 10 equal parts 4 .

Original: Moreover RID is robust to augmentations that do not change the visual content and in this sense is much better than RMSE.
Revised: Moreover RID is robust to augmentations that do not change the visual content and in this sense is much better than RMSE.

Original: To compare new metric with RMSE we train a vanilla VAE with resnet-like architecture on CIFAR10.
Revised: To compare new metric with RMSE, we train a vanilla VAE with resnet-like architecture on CIFAR10.

Original: We compute RID for its reconstructions andfor real images with the augmentation (mirror 10% pad + random crop).
Revised: We compute RID for its reconstructions and real images with the augmentation (mirror 10% pad + random crop).

Original: In Table 2 we show that RMSE for VAE is better in comparison to augmented images (AUG) but we are not satisﬁed with its reconstructions (see Figure 8 in Appendix D.4), Figure 3 provides even more convincing results.
Revised: In Table 2 we show that RMSE for VAE is better in comparison to augmented images (AUG), but we are not satisﬁed with its reconstructions (see Figure 12 in Appendix E.4), Figure 3 provides even more convincing results.

Original: RID allows more adequate comparison, for VAE it is dramatically higher (44.33) than for AUG (1.57).
Revised: RID allows a fair comparison, for VAE it is dramatically higher (44.33) than for AUG (1.57).

Original: Value 1.57 for AUG says that KL divergence is close to zero and thus content is almost not changed.
Revised: Value 1.57 for AUG says that KL divergence is close to zero and thus content is almost not changed.

Original: We also provide estimated RID and RMSE for AGE that was publicly available 4 .
Revised: We also provide estimated RID and RMSE for AGE that was publicly available 5 .

Original: From Table 2 we see that PAGANs outperform AGE which reﬂects that our model has better reconstruction quality.
Revised: From Table 2 we see that PAGANs outperform AGE which reﬂects that our model has better reconstruction quality.

Original: Importance of adversarial loss
Revised: Importance of adversarial loss

Original: To prove the importance of an adversarial loss weperform an experiment replacing adversarial loss with  standard L 1 pixel-wise distance between source images and corresponding reconstructions and compared FID, IS and RID metrics.
Revised: To prove the importance of an adversarial loss, we experiment replacing adversarial loss with the [Table] standard L 1 pixel-wise distance between source images and corresponding reconstructions and compared FID, IS and RID metrics.

Original: Using an augmentation in this setting is ambiguous, thus we did not use any augmentation in training of the changed model.
Revised: Using an augmentation in this setting is ambiguous. Thus we did not use any augmentation in training of the changed model.

Original: Quantitative results for the experiment[Table] are provided in Table 3.
Revised: Quantitative results for the experiment are provided in Table 3.

Original: IS and FID results suggest that our model without adversarial loss performed worse in generation.
Revised: IS and FID results suggest that our model without adversarial loss performed worse in generation.

Original: Reconstruction quality signiﬁcantly dropped considering RID.
Revised: Reconstruction quality signiﬁcantly dropped considering RID.

Original: Visual results in
Revised: Visual results in

Original: Appendix D. conﬁrm 
Revised: Appendix E conﬁrm our quantitative ﬁndings

Original: Importance of augmentation
Revised: Importance of augmentation

Original: In ALICE model (Li et al., 2017) an adversarial reconstruction loss was implemented without an augmentation.
Revised: In ALICE model (Li et al., 2017) an adversarial reconstruction loss was implemented without an augmentation.

Original: As we discussed in Section 1 its absence leads to undesirable effects.
Revised: As we discussed in Section 1 its absence leads to undesirable effects.

Original: Here we run an experiment to show that our model without augmentation performs worse.
Revised: Here we run an experiment to show that our model without augmentation performs worse.

Original: Quantitative results provided in Table 3 illustrate that our model without an augmentation fails to recover both good reconstruction and generation properties.
Revised: Quantitative results provided in Table 3 illustrate that our model without an augmentation fails to recover both good reconstruction and generation properties.

Original: Visual comparisons can be found in Appendix D.2.
Revised: Visual comparisons can be found in Appendix E.2.

Original: Using the results obtained from  last two experiments we conclude that adversarial reconstruction loss works signiﬁcantly better with augmentation. 
Revised: Using the results obtained from the last two experiments we conclude that adversarial reconstruction loss works signiﬁcantly better with augmentation. Choice of Augmentation

Revised: Experiments checking augmentation effects (see Appendix B for details) conclude the following.

Revised: A good augmentation: 1) is required to be non-deterministic, 2) should preserve the content of source image, 3) should be hard to use pixel-wise comparison for discriminator.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper we proposed a novel framework with an augmented adversarial reconstruction loss.
Revised: In this paper, we proposed a novel framework with an augmented adversarial reconstruction loss.

Original: We introduced RID to estimate reconstructions quality for images.
Revised: We introduced RID to estimate reconstructions quality for images.

Original: It was empirically shown that this metric has the ability to perform content-based comparison of reconstructed images.
Revised: It was empirically shown that this metric could perform content-based comparison of reconstructed images.

Original: Using RID we proved the value ofan augmentation in our experiments.
Revised: Using RID, we proved the value of augmentation in our experiments.

Original: We showed that the augmented adversarial loss in this framework plays a key role for getting not only good reconstructions but good generated images.
Revised: We showed that the augmented adversarial loss in this framework plays a key role in getting not only good reconstructions but good generated images.

Original: Some open questions are still left for future work.
Revised: Some open questions are still left for future work.

Original: More complex architectures may be used to achieve better IS and RID.
Revised: More complex architectures may be used to achieve better IS and RID.

Original: The random shift augmentation may not the only possible choice,  other  choices remained undiscovered.
Revised: The random shift augmentation may not the only possible choice, and other smart choices are also possible

Original: A PPENDIX A P ROOFS A. P ROOF OF P ROPOSITION 1 ( OPTIMAL DISCRIMINATOR )
Revised: A PPENDIX A P ROOFS A. P ROOF OF P ROPOSITION 1 ( OPTIMAL DISCRIMINATOR )

Original: Proposition 1.
Revised: Proposition 1.

Original: Given a ﬁxed generator G θ and a ﬁxed encoder E ϕ , the optimal discriminator D ψ ∗ is
Revised: Given a ﬁxed generator G θ and a ﬁxed encoder E ϕ , the optimal discriminator D ψ [Equation] is

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For ﬁxed generator and encoder, the value function V ( ψ ) with respect to the discriminator is
Revised: For ﬁxed generator and encoder, the value function V ( ψ ) with respect to the discriminator is

Original: [Equation]
Revised: [Equation]

Original: Let us introduce new variables and notations
Revised: Let us introduce new variables and notations

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Using the results of the paper Goodfellow et al.
Revised: Using the results of the paper Goodfellow et al.

Original: (2014) we obtain
Revised: (2014) we obtain

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF P ROPOSITION
Revised: A. P ROOF OF P ROPOSITION

Original: Proposition 2.
Revised: Proposition 2.

Original: The minimization of the value function V under an optimal discriminator D ψ ∗ is equivalent to the minimization of the expected Jensen-Shanon divergence between r ( y | x ) and p θ,ϕ ( y | x ) , i.e.
Revised: The minimization of the value function V under an optimal discriminator D ψ ∗ is equivalent to the minimization of the expected Jensen-Shanon divergence between r ( y | x ) and p θ,ϕ ( y | x ) , i.e.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: As in the paper Goodfellow et al.
Revised: As in the paper Goodfellow et al.

Original: (2014) we rewrite the value function V ( θ, ϕ ) for the optimal discriminator D ψ ∗ as follows
Revised: (2014) we rewrite the value function V ( θ, ϕ ) for the optimal discriminator D ψ ∗ as follows

Original: [Equation]
Revised: [Equation]

Revised: A PPENDIX B C HOICE OF A UGMENTATION

Revised: Augmentation choice might be problem speciﬁc.

Revised: Thereby we additionally study different augmentations and provide an intuition how to choose the right transformation.

Revised: Theory suggests to pick up a stochastic augmentation.

Revised: The practical choice should take into account the desired properties of reconstructions.

Revised: A random shift of an image by a small margin is sufﬁcient to create good quality reconstructions.

Revised: However, this shift should not be large because it may inherit augmentation artifacts.

Revised: This can be spotted beforehand just looking at pairs ( x, a ( x )) .

Revised: Once these pairs are not satisfactory, model reconstructions would be bad as well.

Revised: [Figure]

Revised: [Figure]

Revised: B. P ADDING S IZE

Revised: In this experiment

Revised: [Figure]

Revised: we investigate the effects caused by the padding and random crop augmentation.

Revised: We choose different padding size (comparatively to the original image size) and plot FID, RID and IS metrics.

Revised: The results provide the intuition to choose padding size (see Figure 4).

Revised: Padding should be chosen to maintain visual content while making impossible to compare augmented and original images by nearly element-wise comparison.

Revised: Larger padding cause undesirable effects in reconstructions that are captured by RID (see Figure 5).

Revised: Visual quality of samples, on the other hand is slightly better with more aggressive augmentation considering FID metric, what is explained by more robust training due to less mode collapse problem.

Revised: B. O THER A UGMENTATIONS

Revised: We also checked two different augmentation types: Gaussian blur and random contrast (see Figures 6,7).

Revised: Both augmentations led to highly unstable training and did not yield satisfactory results (IS was 2.15 and 4.18 respectively).

Revised: Therefore we conclude that a good augmentation is better to change spatial image structure preserving content (as padding does) what will force the discriminator to take content into account.

Revised: Augmentation

Revised: IS

Revised: [Table]

Revised: The number is taken not from the last epoch to save computational resources and compare results after the same number of epochs.

Revised: A good augmentation:

Revised: [Figure]

Revised: [Figure]

Revised: • is required to be non-deterministic • should preserve the content of source image • should be hard to use pixel-wise comparison for discriminator

Original: A PPENDIX B E XTENDING PAGAN S B. f - DIVERGENCE PAGAN S
Revised: A PPENDIX C E XTENDING PAGAN S C. f - DIVERGENCE PAGAN S

Original: f -GANs (Nowozin et al., 2016) are the generalization of GAN approach.
Revised: f -GANs (Nowozin et al., 2016) are the generalization of GAN approach.

Original: Nowozin et al.
Revised: Nowozin et al.

Original: (2016) introduces the model which minimizes the f -divergence D f (Ali & Silvey, 1966) between the true distribution p ∗ ( x ) and the model distibution p θ ( x ) , i.e. it solves the optimization problem
Revised: (2016) introduces the model which minimizes the f -divergence D f (Ali & Silvey, 1966) between the true distribution p ∗ ( x ) and the model distibution p θ ( x ) , i.e. it solves the optimization problem

Original: [Equation]
Revised: [Equation]

Original:  where f : R + −→ R is a convex, lower-semicontinuous function satisfying f (1) = 0 .
Revised: ) where f : R + −→ R is a convex, lower-semicontinuous function satisfying f (1) = 0 .

Original: The minimax game for f -GANs is deﬁned as
Revised: The minimax game for f -GANs is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: 28) where V ( θ, ψ ) is a value function and f ∗ is a Fenchel conjugate of f (Nguyen et al., 2008).
Revised: (28) where V ( θ, ψ ) is a value function and f ∗ is a Fenchel conjugate of f (Nguyen et al., 2008).

Original: For ﬁxed parameters θ , the optimal T ψ ∗ ( x ) is f (cid:48) (cid:18) p ∗ ( x [Equation] p θ ( x ) (cid:19) .
Revised: For ﬁxed parameters θ , the optimal T ψ ∗ ( x ) is f (cid:48) (cid:18) p ∗ ( x ) p θ ( x ) (cid:19) .

Original: Then the value function V ( θ, ψ ∗ ) for optimal parameters ψ ∗ equals to f -divergence between the distributions p ∗ and p θ (Nguyen et al., 2008), i.e.
Revised: Then the value function V ( θ, ψ ∗ ) for optimal parameters ψ ∗ equals to f -divergence between the distributions p ∗ and p θ (Nguyen et al., 2008), i.e.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We can straightforwardly extend the deﬁnition of PAGANs model to f
Revised: We can straightforwardly extend the deﬁnition of PAGANs model to f

Original: -PAGANs.
Revised: -PAGANs.

Original: We just introduce for each matching problem the f -GAN value function, i.e.
Revised: We just introduce for each matching problem the f -GAN value function, i [Equation] e.

Original: • generator matching:
Revised: • generator matching

Original: [Equation]
Revised: [Equation]

Original: • encoder matching:
Revised: • encoder matching:

Original: [Equation]
Revised: [Equation]

Original: • reconstruction matching:
Revised: • reconstruction matching:

Original: [Equation]
Revised: [Equation]

Original: B. W ASSERSTEIN PAGAN S
Revised: C. W ASSERSTEIN PAGAN S

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: (2017) proposed WGANs model for minimizing the Wasserstein-1 distance between the distributions p ∗ ( x ) and p θ , i.e.
Revised: (2017) proposed WGANs model for minimizing the Wasserstein-1 distance between the distributions p ∗ ( x ) and p θ , i

Original: [Equation]
Revised: [Equation]

Original:  Because the distance W ( p ∗ ( x ) , p θ ( x )) is intractable they consider solving the KantorovichRubinstein dual problem (Villani, 2008)
Revised: e. [Equation] [Equation] Because the distance W ( p ∗ ( x ) , p θ ( x )) is intractable they consider solving the KantorovichRubinstein dual problem (Villani, 2008)

Original: [Equation]
Revised: [Equation]

Original: As in Section B.1 we can easily extend the PAGANs model to WPAGANs.
Revised: As in Section C.1 we can easily extend the PAGANs model to WPAGANs.

Original: In each matching problem the corresponding distance between distributions will be Wasserstein-1 distance.
Revised: In each matching problem the corresponding distance between distributions will be Wasserstein-1 distance.

Original: A PPENDIX C O THER MODELS AND EXPERIMENT DETAILS C. T RAINING W ASSERSTEIN PAGAN
Revised: A PPENDIX D O THER MODELS AND EXPERIMENT DETAILS D. T RAINING W ASSERSTEIN PAGAN

Original: As another concurrent approach to match implicit distributions we can use Wasserstein distance.
Revised: As another concurrent approach to match implicit distributions we can use Wasserstein distance.

Original: Recent empirical works showed promising results (Gulrajani et al., 2017; Gemici et al., 2018) and thus they are interesting to compare with.
Revised: Recent empirical works showed promising results (Gulrajani et al., 2017; Gemici et al., 2018) and thus they are interesting to compare with.

Original: As mentioned above we still need a critic to work on pairs of images.
Revised: As mentioned above we still need a critic to work on pairs of images.

Original: Unlike GAN frameworks it is desirable to have a strong critic.
Revised: Unlike GAN frameworks it is desirable to have a strong critic.

Original: A channel wise concatenation for pairs ( x, y ) worked the best in sense of visual quality and training stability.
Revised: A channel wise concatenation for pairs ( x, y ) worked the best in sense of visual quality and training stability.

Original: As a default choice to improve Wasserstein distance optimization we applied the gradient penalty proposed in Gulrajani et al.
Revised: As a default choice to improve Wasserstein distance optimization we applied the gradient penalty proposed in Gulrajani et al.

Original: To apply the gradient penalty for a critic on pairs we have to interpolate between pairs ( x, y ) and ( x (cid:48) , y (cid:48) ) .
Revised: To apply the gradient penalty for a critic on pairs we have to interpolate between pairs ( x, y ) and ( x (cid:48) , y (cid:48) ) .

Original: There are still two choices:
Revised: There are still two choices:

Original: • shared alpha
Revised: • shared alpha

Original: [Equation]
Revised: [Equation]

Original: • independent alpha for each part
Revised: • independent alpha for each part

Original: [Equation]
Revised: [Equation]

Original: Empirically we found no differences in results and in further experiments used shared alpha as a default choice.
Revised: Empirically we found no differences in results and in further experiments used shared alpha as a default choice.

Original: The gradient penalty strength parameter λ was set to 10 as recommended by Gulrajani et al.
Revised: The gradient penalty strength parameter λ was set to 10 as recommended by Gulrajani et al.

Original: We used 10 discriminator steps per 1 generator/encoder step for WPAGAN to slightly improve quality in this setting, other parameters were unchanged.
Revised: We used 10 discriminator steps per 1 generator/encoder step for WPAGAN to slightly improve quality in this setting, other parameters were unchanged.

Original: In Table 4 we present results for
Revised: In Table 5 we present results for

Original: Wasserstein loss used instead of standard GAN objective in PAGAN model.
Revised: Wasserstein loss used instead of standard GAN objective in PAGAN model.

Original: While having good reconstructions this type of loss failed to achieve good generation results. 
Revised: While having good reconstructions this type of loss failed to achieve good generation results. Model

Original: [Table]
Revised: [Table]

Revised: A PPENDIX E I MAGES

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: D. PAGAN V ISUAL R ESULTS
Revised: E. PAGAN V ISUAL R ESULTS

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original:  FOR R ECONSTRUCTION I NCEPTION S 
Revised: VAE FOR R ECONSTRUCTION I NCEPTION S CORE

Original: [Figure]