Original: D OUBLY S TOCHASTIC A DVERSARIAL A UTOENCODER
Revised: D OUBLY S TOCHASTIC A DVERSARIAL A UTOENCODER

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Any autoencoder network can be turned into a generative model by imposing an arbitrary prior distribution on its hidden code vector.
Revised: Any autoencoder network can be turned into a generative model by imposing an arbitrary prior distribution on its hidden code vector.

Original: Variational Autoencoder uses a KL divergence penalty to impose the prior, whereas Adversarial Autoencoder uses generative adversarial networks.
Revised: Variational Autoencoder uses a KL divergence penalty to impose the prior, whereas Adversarial Autoencoder uses generative adversarial networks.

Original: A straightforward modiﬁcation of Adversarial
Revised: A straightforward modiﬁcation of Adversarial

Original: Autoencoder is to replace the adversary by maximum mean discrepancy (MMD) test.
Revised: Autoencoder is to replace the adversary by maximum mean discrepancy (MMD) test.

Original: This replacement leads to a new type of probabilistic autoencoder, which is also discussed in our paper.
Revised: This replacement leads to a new type of probabilistic autoencoder, which is also discussed in our paper.

Original: However, an essential challenge remains in both of these probabilistic autoencoders, namely that the only source of randomness at the output of encoder, is the training data itself.
Revised: However, an essential challenge remains in both of these probabilistic autoencoders, namely that the only source of randomness at the output of encoder, is the training data itself.

Original: Lack of enough stochasticity can make the optimization problem nontrivial.
Revised: Lack of enough stochasticity can make the optimization problem nontrivial.

Original: As a result, they can lead to degenerate solutions where the generator collapses into sampling only a few modes.
Revised: As a result, they can lead to degenerate solutions where the generator collapses into sampling only a few modes.

Original: Our proposal is to replace the adversary of adversarial autoencoder by a space of stochastic functions.
Revised: Our proposal is to replace the adversary of adversarial autoencoder by a space of stochastic functions.

Original: This replacement introduces a a new source of randomness, which can be considered as a continuous control for encouraging explorations .
Revised: This replacement introduces a a new source of randomness, which can be considered as a continuous control for encouraging explorations .

Original: This prevents the adversary from ﬁtting too closely to the generator and therefore leads to more diverse set of generated samples.
Revised: This prevents the adversary from ﬁtting too closely to the generator and therefore leads to more diverse set of generated samples.

Original: Consequently, the decoder serves as a better generative network, which unlike MMD nets scales linearly with the amount of data.
Revised: Consequently, the decoder serves as a better generative network, which unlike MMD nets scales linearly with the amount of data.

Original: We provide mathematical and empirical evidence on how this replacement outperforms the pre-existing architectures.
Revised: We provide mathematical and empirical evidence on how this replacement outperforms the pre-existing architectures.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Any autoencoder network can be turned into a generative model by imposing an arbitrary prior distribution on its hidden code vector.
Revised: Any autoencoder network can be turned into a generative model by imposing an arbitrary prior distribution on its hidden code vector.

Original: Variational Autoencoder (VAE) [2] uses a KL divergence penalty to impose the prior on the hidden code vector of the autoencoder, whereas Adversarial
Revised: Variational Autoencoder (VAE) [2] uses a KL divergence penalty to impose the prior on the hidden code vector of the autoencoder, whereas Adversarial

Original: Autoencoder (AAE) [1] uses generative adversarial networks (GAN) [2].
Revised: Autoencoder (AAE) [1] uses generative adversarial networks (GAN) [2].

Original: GAN trades the complexities of sampling algorithms with the complexities of searching Nash equilibrium in minimax games.
Revised: GAN trades the complexities of sampling algorithms with the complexities of searching Nash equilibrium in minimax games.

Original: Such minimax architectures get updated directly with the help of data examples and gradients ﬂowing through a generative and an adversary.
Revised: Such minimax architectures get updated directly with the help of data examples and gradients ﬂowing through a generative and an adversary.

Original: A straightforward modiﬁcation of AAE is to replace the adversarial network by maximum mean discrepancy (MMD) net [4-5].
Revised: A straightforward modiﬁcation of AAE is to replace the adversarial network by maximum mean discrepancy (MMD) net [4-5].

Original: This replacement leads to a new type of probabilistic autoencoder, which is also discussed in our paper.
Revised: This replacement leads to a new type of probabilistic autoencoder, which is also discussed in our paper.

Original: However, an essential challenge remains in both of these probabilistic autoencoders, namely that the only source of randomness at the output of encoder, is the training data itself.
Revised: However, an essential challenge remains in both of these probabilistic autoencoders, namely that the only source of randomness at the output of encoder, is the training data itself.

Original: Lack of enough stochasticity can make the minimax search non-trivial.
Revised: Lack of enough stochasticity can make the minimax search non-trivial.

Original: As a result, it can lead to degenerate solutions where the generator collapses into sampling only a few modes.
Revised: As a result, it can lead to degenerate solutions where the generator collapses into sampling only a few modes.

Original: In order to mitigate the mode collapse issue, we introduce randomness to the loss functions of such minimax architectures, which can be considered as a continuous control for encouraging explorations .
Revised: In order to mitigate the mode collapse issue, we introduce randomness to the loss functions of such minimax architectures, which can be considered as a continuous control for encouraging explorations .

Original: This prevents the adversary from ﬁtting too closely to the generator and therefore leads to more diverse set of generated samples.
Revised: This prevents the adversary from ﬁtting too closely to the generator and therefore leads to more diverse set of generated samples.

Original: Introducing randomness at the high dimensional feature space can make the minimax search problem very challenging.
Revised: Introducing randomness at the high dimensional feature space can make the minimax search problem very challenging.

Original: Fortunately, this is a much easier search problem in our case, thanks to the lower dimensionality of latent space (encoder output).
Revised: Fortunately, this is a much easier search problem in our case, thanks to the lower dimensionality of latent space (encoder output).

Original:  P ROBABILISTIC A UTOENCODERS
Revised:  P ROBABILISTIC A UTOENCODERS

Original: We consistently refer to any random variable and their realization with calligraphic fonts and small letters respectively.
Revised: We consistently refer to any random variable and their realization with calligraphic fonts and small letters respectively.

Original: Fix an autoencoder (AE) with encoder parameters of θ ∈ Θ that gets i.i.d data samples X , X 1 , ..., X N as input and outputs the corresponding latent vectors Z , Z 1 , ..., Z N .
Revised: Fix an autoencoder (AE) with encoder parameters of θ ∈ Θ that gets i.i.d data samples X , X 1 , ..., X N as input and outputs the corresponding latent vectors Z , Z 1 , ..., Z N .

Original: Assume we don’t know the distribution of X
Revised: Assume we don’t know the distribution of X

Original: but we want to impose an arbitrary distribution P on Z .
Revised: but we want to impose an arbitrary distribution P on Z .

Original: This is because we like to think of an decoder as a generative network with prior P .
Revised: This is because we like to think of the decoder as a generative network with prior P .

Original: The encoding function Q θ ( Z|X ) induces aggregated posterior distribution Q θ ( Z ) formulated in eq. 1:
Revised: The encoding function Q θ ( Z|X ) induces aggregated posterior distribution Q θ ( Z ) formulated in eq. 1:

Original: [Equation]
Revised: [Equation]

Original: The decoder and encoder networks of a standard AE can be trained together by minimizing the difference between the original data and its reconstruction.
Revised: The decoder and encoder networks of a standard AE can be trained together by minimizing the difference between the original data and its reconstruction.

Original: We can impose P on the latent code vector through regularization of a standard AE by minimizing the discrepancies between aggregated posterior Q θ ( Z ) and an arbitrary prior P .
Revised: We can impose P on the latent code vector through regularization of a standard AE by minimizing the discrepancies between aggregated posterior Q θ ( Z ) and an arbitrary prior P .

Original: This can be formalized as the following optimization problem:
Revised: This can be formalized as the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where δ is a suitable discrepancy measure.
Revised: where δ is a suitable discrepancy measure.

Original: In other words, parameter θ should be optimized s.t the transported randomness from X match up to that of prior P .
Revised: In other words, parameters of the encoder θ should be optimized s.t the transported randomness from X match up to that of prior P .

Original: Achievement of a solution for eq. 2 via deterministic functions is guaranteed given enough stochasticity in X , thanks to the Lemma 2.22 in [11].
Revised: Achievement of a solution for eq. 2 via deterministic functions is guaranteed given enough stochasticity in X , thanks to the Lemma 2.22 in [11].

Original: However, lack of enough stochasticity in X leads to not smooth enough aggregated posterior Q θ ( Z ) which makes the minimization in Eq.
Revised: However, lack of enough stochasticity in X leads to not smooth enough aggregated posterior Q θ ( Z ) which makes the minimization in Eq.

Original: 2 non-trivial.
Revised: 2 non-trivial.

Original: This is specially important considering the deterministic functionality of encoder.
Revised: This is specially important considering the deterministic functionality of encoder.

Original: Thus us because the only source of randomness originates from the training data itself, which may not be enough for smoothing out the aggregated posterior.
Revised: It is because the only source of randomness originates from the training data itself, which may not be enough for smoothing out the aggregated posterior.

Original: 2. I NTRODUCING R ANDOMNESS
Revised: 2. I NTRODUCING R ANDOMNESS

Original: Some ways of introducing randomness for smoothing out the aggregated posterior Q θ ( Z ) include:
Revised: Some ways of introducing randomness for smoothing out the aggregated posterior Q θ ( Z ) include:

Original: • Introducing the stochasticity N at the input or output of the encoder function:
Revised: • Introducing the stochasticity N at the input or output of the encoder function:

Original: [Equation]
Revised: [Equation]

Original: This may ask for re-parametrization tricks [2].
Revised: This may ask for re-parametrization tricks [2].

Original: • Dropouts [9]
Revised: • Dropouts [9]

Original: [Equation]

Original: Different ways of introducing and exploiting randomness lead to different training dynamics.
Revised: Different ways of introducing and exploiting randomness lead to different training dynamics.

Original: The choice of discrepancy measure δ itself leads to various types of training dynamics.
Revised: The choice of discrepancy measure δ itself leads to various types of training dynamics.

Original: Our proposal is to introduce randomness at the heart of the discrepancy measure itself.
Revised: Our proposal is to introduce randomness at the heart of the discrepancy measure itself.

Original: First, we discuss two known discrepancy measures in the next section.
Revised: First, we discuss two known discrepancy measures in the next section.

Original: 2. D ISCREPANCY M EASURE
Revised: 2. D ISCREPANCY M EASURE

Original: Adversarial autoencoder can be recovered with the following Jenson-Shannon divergence (JSD) choice for the discrepancy measure:
Revised: Adversarial autoencoder can be recovered with the following Jenson-Shannon divergence (JSD) choice for the discrepancy measure:

Original: [Equation]
Revised: [Equation]

Original: where Y ∼ P and ˜ Y ∼ Q θ ( Z ) and D
Revised: where Y ∼ P and ˜ Y ∼ Q θ ( Z ) and D

Original: α is the adversary with parameter α .
Revised: α is the adversary with parameter α .

Original: Moreover E is w.r.t to the associated random variable of each term.
Revised: Moreover E is w.r.t to the associated random variable of each term.

Original: A straightforward modiﬁcation is to replace the adversarial network by maximum mean discrepancy (MMD) net [4-5] by assuming the following discrepancy measure:
Revised: A straightforward modiﬁcation is to replace the adversarial network by maximum mean discrepancy (MMD) net [4-5] by assuming the following discrepancy measure:

Original: [Equation]
Revised: [Equation]

Original: where f is a function living in a reproducing kernel Hilbert space RKHS H and Y ∼ P and ˜
Revised: where f is a function living in a reproducing kernel Hilbert space RKHS H and Y ∼ P and ˜

Original: Y ∼ Q θ ( Z ) as above.
Revised: Y ∼ Q θ ( Z ) as above.

Original: Due to the reproducing property of H , the expectation of any function f in RKHS H with respect to random variable Y can be computed as an inner product with its so called kernel mean embedding E [ k [Equation] Y [Equation]
Revised: Due to the reproducing property of H , the expectation of any function f in RKHS H with respect to random variable Y can be computed as an inner product with its so called kernel mean embedding E [ k ( Y , . )] :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: k being the kernel associated with RKHS H .
Revised: k being the kernel associated with RKHS H .

Original: Deﬁnition 

Revised: Deﬁnition

Original: A kernel k ( x, x (cid:48) ) :
Revised: A kernel k ( x, x (cid:48) ) :

Original: X × X
Revised: X × X

Original: → R is positive deﬁnite (PD) when for all n > 1 and x 1 , x 2 , .., x n ∈ X and c 1 , ..., c n ∈ R , we have (cid:80) i,j c i c j k ( x i , x j ) ≥ 0 .
Revised: → R is positive deﬁnite (PD) when for all n > 1 and x 1 , x 2 , .., x n ∈ X and c 1 , ..., c n ∈ R , we have (cid:80) i,j c i c j k ( x i , x j ) ≥ 0 .

Original: Gretton [12] demonstrates that a closed form solution exists for Eq.
Revised: Gretton [12] demonstrates that a closed form solution exists for Eq.

Original: 6 with the following unbiased estimator :
Revised: 6 with the following unbiased estimator :

Original: [Equation]
Revised: [Equation]

Revised: (cid:88) m

Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: k

Revised: [Equation]

Revised: We refer to a probabilistic AE that uses Eq.

Revised: 8 as discrepancy measure MMD-AE.

Original:  D OUBLY S TOCHASTIC K ERNEL M ACHINES AS A DVERSARY
Revised:  D OUBLY S TOCHASTIC K ERNEL M ACHINES AS A DVERSARY

Original: The existence of closed form expression in Eq.
Revised: The existence of closed form expression in Eq.

Original: 8 makes the implementation of MMD nets straightforward [4-5], however, it hides the minimax nature of the problem.
Revised: 8 makes the implementation of MMD-AE straightforward, however, it hides the minimax nature of the problem.

Original: If we replace eq. 6 in eq.
Revised: If we replace eq. 6 in eq.

Original: 2 we are back to a minimax problem similar to that of adversarial networks as the following:
Revised: 2, we are back to a minimax problem similar to that of adversarial networks as the following:

Original: [Equation]
Revised: [Equation]

Original: Here is where the discussed problem with the lack of enough stochasticity of X and non-smooth aggregated posterior gets more clear.
Revised: Here is where the discussed problem with the lack of enough stochasticity of X and non-smooth aggregated posterior gets more clear.

Original: Using Eq.
Revised: Using Eq.

Original: 7 and Eq.
Revised: 7 and Eq.

Original: 9 we note that adversary’s best response training dynamic is determined by the stochastic gradient terms ∂δ∂f = ξ ( . ) :
Revised: 9, we note that adversary’s best response training dynamic is determined by the stochastic gradient terms ∂δ∂f = ξ ( . ) :

Original: =
Revised: =

Original: E [ k ( Y , . )
Revised: E [ k ( Y , . )

Original: − k ( ˜ Y , . )] .
Revised: − k ( ˜ Y , . )] .

Original: Since prior P is smooth enough, term
Revised: Since prior P is smooth enough, term

Original: [Equation]
Revised: [Equation]

Original: [ k [Equation] Y , . )] determines the smoothness of the stochastic gradients.
Revised: , . )] determines the smoothness of the stochastic gradients.

Original: Not enough stochasticity in X implies non-smooth gradients and therefore more difﬁcult optimization problem.
Revised: Not enough stochasticity in X implies non-smooth gradients and therefore more difﬁcult optimization problem.

Original: We propose to massage the stochastic gradients ξ ( . ) with another source of stochasticity W .
Revised: We propose to massage the stochastic gradients ξ ( . ) with extra source of stochasticity W .

Original: This leads to doubly stochastic gradient terms ζ ( . ) :
Revised: This leads to doubly stochastic gradient terms ζ ( . ) :

Original: =
Revised: =

Original: E W ξ ( . ) .
Revised: E W ξ ( . ) .

Original: Introducing new source of randomness to ξ ( . ) becomes feasible thanks to the existing duality between the Kernel and
Revised: Introducing new source of randomness to ξ ( . ) becomes feasible thanks to the existing duality between the Kernel and

Original: Random processes according the following Theorem:
Revised: Random processes according the following Theorem:

Original: Theorem 1  Duality between Kernels and Random Processes : If k ( x, x (cid:48) ) is a PD kernel, then there exits a set Ω , a measure P on Ω , and random function φ W ( x ) :
Revised: Theorem 1 [15] Duality between Kernels and Random Processes : If k ( x, x (cid:48) ) is a PD kernel, then there exits a set Ω , a measure P on Ω , and random function φ W ( x ) :

Original: X → R from L 2 (Ω , P ) , such thatk ( [Equation] P ( W
Revised: X → R from L 2 (Ω , P ) , such that

Original: [Equation]
Revised: [Table]

Original: Example For Gaussian RBF kernel,

Original: [Equation]
Revised: [Figure]

Original: and P ( w

Original: [Equation] As the result of Theorem 1, we can rewrite the massaged gradients as ζ = [ φ W ( Y ) − φ W ( ˜ Y )] φ W ( . ) with W ∼ P ( W ) . [Figure] =
Revised: Not enough stochasticity in the X limits the adversary power as is reﬂected in its stochastic gradient terms ξ =

Original: E
Revised: E

Original: Y∼ Q θ ( Z )
Revised: Y∼ Q θ ( Z )

Original: [ k ( Y , . )] .
Revised: [ k ( Y , . )] .

Original: With the introduction of new source of randomness, the adversary gains extra power as is reﬂected in its doubly stochastic gradient terms gradient ζ = E W ξ ( . ) .
Revised: With the introduction of new source of randomness, the adversary gains extra power, as is reﬂected in its doubly stochastic gradient terms gradient ζ = E W ξ ( . ) .

Original: This extra boost to adversary however can lead to the search outside of H where the gradients are no longer valid closed form expressions.
Revised: This extra boost to adversary, however, can lead to the search outside of H where the gradients are no longer valid closed form expressions.

Original: Fortunately [6] shows that with small learning rates, the search returns back to H and convergence is guaranteed.
Revised: Fortunately, [6] shows that with small learning rates, the search returns back to H and convergence is guaranteed.

Revised: Example For Gaussian RBF kernel, k ( x − x (cid:48) )

Revised: =

Revised: exp( (cid:107) x − x (cid:48) (cid:107) 2 / 2 σ 2 ) , φ W ( x ) = exp( − i W (cid:62) x ) and P ( w ) =

Revised: [Equation]

Revised: As the result of Theorem 1, we can rewrite the massaged gradients as ζ = [ φ W ( Y ) − φ W ( ˜ Y )] φ W ( . ) with W ∼ P ( W ) .

Original: 3. A PPROXIMATING ADVERSARY f BY ITS D OUBLY S TOCHASTIC G RADIENT T ERMS
Revised: 3. A PPROXIMATING ADVERSARY f BY ITS D OUBLY S TOCHASTIC G RADIENT T ERMS

Original: Using a similar approach to Doubly Stochastic Kernel Machines [6], we can now approximate a new adversary with parameter α by the linear combination of doubly stochastic gradient terms ζ i.e. f α ( . )
Revised: Using a similar approach to Doubly Stochastic Kernel Machines [6], we can now approximate a new adversary with parameter α by the linear combination of doubly stochastic gradient terms ζ i.e. f α ( . )

Original: =
Revised: =

Original: αζ ( . ) .
Revised: αζ ( . ) .

Original: However we modify the adversary’s objective function to − [ φ W ( ˜ Y ) φ W ( . )]
Revised: However, we modify the adversary’s objective function to − [ φ W ( ˜ Y ) φ W ( . )]

Original: ˜ Y∼ Q θ ( Z ) .
Revised: ˜ Y∼ Q θ ( Z ) .

Original: This modiﬁcation does not affect the ﬁxed points of the best response dynamics but provides much stronger gradients early in learning.
Revised: This modiﬁcation does not affect the ﬁxed points of the best response dynamics but provides much stronger gradients early in learning.

Original: GAN [3] applies the same modiﬁcation to the objective function of the adversary.
Revised: GAN [3] applies the same modiﬁcation to the objective function of the adversary.

Original: The adversary affects the dynamic of training though the adjustment in α .
Revised: The adversary affects the dynamic of training though the adjustment in α .

Original: We refer this probabilistic autoencoder as Doubly Stochastic AAE (DS-AAE). 
Revised: We refer this probabilistic autoencoder as Doubly Stochastic AAE (DS-AAE). [Figure]

Original: Beside the residual error of approximating f with its ﬁrst order gradient terms, there is another concern that needs to be addressed.
Revised: Beside the residual error of approximating f with its ﬁrst order gradient terms, there is another concern that needs to be addressed.

Original: The key difference ζ ( . ) and ξ ( . ) is that ζ ( . ) could fall outside of the RKHS but ξ ( . ) is always in RKHS.
Revised: The key difference ζ ( . ) and ξ ( . ) is that ζ ( . ) could fall outside of the RKHS but ξ ( . ) is always in RKHS.

Original: This is due to the term φ w ( . )
Revised: This is due to the term φ w ( . )

Original: not being in RKHS.
Revised: not being in RKHS.

Original: Fortunately proof of the convergence with small enough learning rate is discussed in [6].
Revised: Fortunately proof of the convergence with small enough learning rate is discussed in [6].

Revised: The scheme of DS-AAE is visualized in Fig.

Original: To sum up, we proposed replacing the adversary with a space H (cid:48) of stochastic functions which may not reside in RKHS H but their convergence to the optimal functions in H is guaranteed.
Revised: To sum up, we proposed replacing the adversary with a space H (cid:48) of stochastic functions, which may not reside in RKHS H but their convergence to the optimal functions in H is guaranteed.

Original: This replacement introduces a new source of randomness which can be considered as a continuous control
Revised: This replacement introduces a new source of randomness,

Original: [Figure]
Revised: [Table]

Original:  for encouraging explorations.
Revised: which can be considered as a continuous control for encouraging explorations.

Original: It prevents the adversary from ﬁtting too closely to the generator and therefore leads to more diverse set of generated samples.
Revised: It prevents the adversary from ﬁtting too closely to the generator and therefore leads to more diverse set of generated samples.

Original: Consequently, the decoder serves as a better generative network which unlike MMD nets scales linearly with the amount of data.
Revised: Consequently, the decoder serves as a better generative network which unlike MMD-AE scales linearly with the amount of data.

Original: We provide empirical evidence on how this replacement outperforms the pre-existing architectures in the next section.[Table]
Revised: We provide empirical evidence on how this replacement outperforms the pre-existing architectures in the next section.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We generate W according to Example 1 with a ﬁxed seed.
Revised: We generate W according to Example 1 with a ﬁxed seed.

Original: The encoder and decoder both havelayers of 1024, 512, 216 hidden units with ReLU activation for every layer except the last layer of decoder that is a sigmoid activation function.
Revised: The encoder and decoder both havelayers of 1024, 512, 216 hidden units with ReLU activation for every layer except the last layer of decoder that is a sigmoid activation function.

Original: Cross entropy is used for reconstruction loss.
Revised: Cross entropy is used for reconstruction loss.

Original: The mini-batch size is 1000.
Revised: The mini-batch size is 1000.

Original: The prior P is Gaussian and the dimensionality of the hidden code is 6 for the DS-AAE and 4 for MD-AAE.
Revised: The prior P is Gaussian and the dimensionality of the hidden code is 6 for the DS-AAE and 4 for MMD-AE.

Original: The only used dropout is at the ﬁrst input layer of the encoder with the rate of 20% .
Revised: The only used dropout is at the ﬁrst input layer of the encoder with the rate of 20% .

Original: Initial learning rate for the reconstruction loss is adjusted at 0.001 and 0.001 for the adversarial architectures, followed by Adam stochastic optimization [13].
Revised: Initial learning rate for the reconstruction loss is adjusted at 0.001 and 0.001 for the adversarial architectures, followed by Adam stochastic optimization [13].

Original: Comparison of deep generative models is hard especially for the log-likelihood-free models [7].
Revised: Comparison of deep generative models is hard especially for the log-likelihood-free models [7].

Original: Parzen window estimation of the log-likelihood is obtained by drawing 10K samples from the trained model on MNIST.
Revised: Parzen window estimation of the log-likelihood is obtained by drawing 10K samples from the trained model on MNIST.

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: From the qualitative perspective, we can see from Fig.
Revised: From the qualitative perspective, we can see from Fig.

Original: 2b, Fig. 2d and reported results in [1] that the drawn samples for both MMD-AAE and AAE are more homogenous than DS-AAE.
Revised: 2b, Fig. 2d and reported results in [1] that the drawn samples for both MMD-AE and AAE are more homogenous than DS-AAE.

Original: In the case of DS-AAE, it is almost as if different persons were writing the digits in each panel.
Revised: In the case of DS-AAE, it is almost as if different persons were writing the digits in each panel.

Original: This quality test is also used in [8].
Revised: This quality test is also used in [8].

Original: This is because DS-AAE enjoys from extra randomness in the minimax optimization framework, which helps the generative model to explore multiple modes and mitigate the risk of collapse.
Revised: This is because DS-AAE enjoys from extra randomness in the minimax optimization framework, which helps the generative model to explore multiple modes and mitigate the risk of collapse.

Original: The learned coding space of DS-AAE exhibits sharp transitions and has no “holes”, as is shown in Fig.
Revised: The learned coding space of DS-AAE exhibits sharp transitions and has no “holes”, as is shown in Fig.

Original: 2 (a).
Revised: 2 (a).

Original: This is important to ensure that generating from any part of prior space results in meaningful samples.
Revised: This is important to ensure that generating from any part of prior space results in meaningful samples.

Original: From this perspective, DS-AAE is similar to AAE and unlike VAE.
Revised: From this perspective, DS-AAE is similar to AAE and unlike VAE.

Original: However it recovers more of a mixture model rather than a 2D-Gaussian.
Revised: However it recovers more of a mixture of 2D-Gaussians rather than a 2D-Gaussian distribution.

Original: We leave further investigation of this interesting observation to future version of this paper.
Revised: We leave further investigation of this interesting observation to future version of this paper.

Original: SVHN experiment is carried out by the the same architecture and batch size.
Revised: SVHN experiment is carried out by the the same architecture and batch size.

Original: However, we have used batch-normalization [14] in all the autoencoder layers including the Softmax layer and number of latent codes is 10 in this case.
Revised: However, we have used batch-normalization [14] in all the autoencoder layers including the Softmax layer and number of latent codes is 10 in this case.

Original: We observe that the generated samples from DS-AA are more diverse, yet they are blurrier than those of  AAE and  GAN architectures.
Revised: We observe that the generated samples from DS-AA are more diverse, yet they are blurrier than those of MMD-AE, AAE and the rest of GAN architectures.

Original: Results are shown in Fig.3.
Revised: Results are shown in Fig.3.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The recent proposal of generative adversarial models, trades the complexities of sampling algorithms with the complexities of searching Nash equilibrium in minimax games.
Revised: The recent proposal of generative adversarial models, trades the complexities of sampling algorithms with the complexities of searching Nash equilibrium in minimax games.

Original: From this vantage point, role of a machine learning researcher is similar to that of a mechanism designer , who intervenes with the dynamic of game through design of suitable loss function.
Revised: From this vantage point, role of a machine learning researcher is similar to that of a mechanism designer , who intervenes with the dynamic of game through design of suitable loss function.

Original: A recent development of probabilistic autoencoder enjoys this framework by placing such minimax games at the output of an encoder to force a desirable prior.
Revised: A recent development of probabilistic autoencoder enjoys this framework by placing such minimax games at the output of an encoder to force a desirable prior.

Original: We propose a different minimax game and loss functions.
Revised: We propose a different minimax game and loss function.

Original: As a mechanism designer, we intervene with the dynamic of training through introducing suitable randomness to the loss functions of such minimax games.
Revised: As a mechanism designer, we intervene with the dynamic of training through introducing suitable randomness to the loss functions of such minimax games.

Original: This makes the game end up in better regions and as a result, better probabilistic autoencoders can be achieved.
Revised: This makes the game end up in better regions and as a result, better probabilistic autoencoders can be achieved.