Original: From em -Projections to Variational Auto-Encoder
Revised: From em -Projections to Variational Auto-Encoder

Original: Abstract
Revised: Abstract

Original: This paper reviews the em -projections in information geometry and the recent understanding of variational auto-encoder, and explains that they share a common formulation as joint minimization of the Kullback-Leibler divergence between two manifolds of probability distributions, and the joint minimization can be implemented by alternating projections or alternating gradient descent
Revised: This paper reviews the em -projections in information geometry and the recent understanding of variational auto-encoder, and explains that they share a common formulation as joint minimization of the Kullback-Leibler divergence between two manifolds of probability distributions, and the joint minimization can be implemented by alternating projections or alternating gradient descent

Original:  Introduction
Revised:  Introduction

Original: Generative modeling is a principled framework for unsupervised learning from unlabeled data as well as semi-supervised learning from both unlabeled and labeled data.
Revised: Generative modeling is a principled framework for unsupervised learning from unlabeled data as well as semi-supervised learning from both unlabeled and labeled data.

Original: Taking advantage of the approximation capacity of deep networks [13, 18, 19], deep generative models have achieved impressive successes in unsupervised and semi-supervised learning.
Revised: Taking advantage of the approximation capacity of deep networks [13, 18, 19], deep generative models have achieved impressive successes in unsupervised and semi-supervised learning.

Original: A popular deep generative model is the generator model [10, 17], which assumes that the observed example is generated by a latent vector via a top-down network, and the latent vector follows a known prior distribution.
Revised: A popular deep generative model is the generator model [10, 17], which assumes that the observed example is generated by a latent vector via a top-down network, and the latent vector follows a known prior distribution.

Original: The top-down network can be interpreted as a decoder, and it is usually jointly trained with a bottom-up inference network that can be interpreted as an encoder in the framework of variational auto-encoder (VAE)
Revised: The top-down network can be interpreted as a decoder, and it is usually jointly trained with a bottom-up inference network that can be interpreted as an encoder in the framework of variational auto-encoder (VAE)

Original: Generative models can be understood within the framework of information geometry, where each probability distribution is treated as a point, and different families of probability distributions form different manifolds.
Revised: Generative models can be understood within the framework of information geometry, where each probability distribution is treated as a point, and different families of probability distributions form different manifolds.

Original: Information geometry characterizes the geometric structures of manifolds of probability distributions, and has been widely applied to statistics, information science, dynamic systems, etc.
Revised: Information geometry characterizes the geometric structures of manifolds of probability distributions, and has been widely applied to statistics, information science, dynamic systems, etc.

Original: Projections from a probability distribution to a manifold can be formulated as minimizing certain divergence between probability distributions, such as the commonly used
Revised: Projections from a probability distribution to a manifold can be formulated as minimizing certain divergence between probability distributions, such as the commonly used

Original: Kullback-Leibler (KL) divergence [7].
Revised: Kullback-Leibler (KL) divergence [7].

Original: This paper reviews the information geometric em -algorithm which is realized by projections along e -geodesic and m -geodesic alternatively [8, 2].
Revised: This paper reviews the information geometric em -algorithm which is realized by projections along e -geodesic and m -geodesic alternatively [8, 2].

Original: We shall also review the recent understanding of variational auto-encoder (VAE) [12].
Revised: We shall also review the recent understanding of variational auto-encoder (VAE) [12].

Original: We explain that em -projections and VAE share a common formulation as joint minimization of the Kullback-Leibler divergence between two manifolds of probability distributions, and the joint minimization can be implemented by alternating projections or alternating gradient descent.
Revised: We explain that em -projections and VAE share a common formulation as joint minimization of the Kullback-Leibler divergence between two manifolds of probability distributions, and the joint minimization can be implemented by alternating projections or alternating gradient descent.

Original:  em -Projections
Revised:  em -algorithm

Original: Consider the probability model p θ ( x, z ) with trainable parameters θ , where x is the observed example, and z is the latent vector.
Revised: Consider the probability model p θ ( x, z ) with trainable parameters θ , where x is the observed example, and z is the latent vector.

Original: Let P = { p θ ( x, z ) , ∀ θ } be the manifold formed by the model.
Revised: Let P = { p θ ( x, z ) , ∀ θ } be the manifold formed by the model.

Original: We may call
Revised: We may call

Original: P the model manifold.
Revised: P the model manifold.

Original: The information geometric em -algorithm considers the class of exponential family model
Revised: The information geometric em -algorithm considers the class of exponential family model

Original: [Equation]
Revised: [Equation]

Original: where s ( x, z ) denotes the sufﬁcient statistics, λ denotes the natural parameters, and Z ( λ ) is the normalizing constant.
Revised: where s ( x, z ) denotes the sufﬁcient statistics, λ denotes the natural parameters, and Z ( λ ) is the normalizing constant.

Original: It is assumed that p θ ( x, z ) =
Revised: It is assumed that p θ ( x, z ) =

Original: p ( x, z ; λ ( θ )) for some parametrization of λ , where λ ( θ ) may not be invertible, and the resulting model is a curved exponential family model.
Revised: p ( x, z ; λ ( θ )) for some parametrization of λ , where λ ( θ ) may not be invertible, and the resulting model is a curved exponential family model.

Original: The exponential family model (1) can also be parametrized by its mean parameters µ ( λ ) =
Revised: The exponential family model (1) can also be parametrized by its mean parameters µ ( λ ) =

Original: E p λ
Revised: E p λ

Original: [ s ( x, z )] , which is a function of λ .
Revised: [ s ( x, z )] , which is a function of λ .

Original: Assuming µ ( λ ) is invertible, we can write p ( x, z | µ ) =
Revised: Assuming µ ( λ ) is invertible, we can write p ( x, z | µ ) =

Original: p ( x, z ; λ ) if µ = µ ( λ ) .
Revised: p ( x, z ; λ ) if µ = µ ( λ ) .

Original: Note here we use the generic notation p , but we write p θ , p ( · ; λ ) and p ( · | µ ) in different styles to distinguish the three different parametrizations.
Revised: Note here we use the generic notation p , but we write p θ , p ( · ; λ ) and p ( · | µ ) in different styles to distinguish the three different parametrizations.

Original: Deﬁne Q
Revised: Deﬁne Q

Original: = { p ( x, z | µ ) , µ = s ( x obs , z im ) , ∀ z im } where x obs is ﬁxed at the observed value of x , and z im is the potential imputed value of z
Revised: = { p ( x, z | µ ) , µ = s ( x obs , z im ) , ∀ z im } where x obs is ﬁxed at the observed value of x , and z im is the potential imputed value of z

Original: The information geometric em -algorithm
Revised: The information geometric em -algorithm

Original: [2, 3] solves the following joint minimization problem:
Revised: [2, 3] solves the following joint minimization problem:

Original: [Equation]
Revised: [Equation]

Original: where D KL ( q (cid:107) p ) =
Revised: where D KL ( q (cid:107) p ) =

Original: E q [log( q/p )] .
Revised: E q [log( q/p )] .

Original: The algorithm is formulated as follows: initialize at a p ∈ P , and alternate the following two projections (see elementary exposition by [21]):
Revised: The algorithm is formulated as follows: initialize at a p ∈ P , and alternate the following two projections (see elementary exposition by [21]):

Original: • e -projection : e -project the current p to Q by minimizing D KL ( q (cid:107) p ) over q ∈ Q .
Revised: • e -projection : e -project the current p to Q by minimizing D KL ( q (cid:107) p ) over q ∈ Q .

Original: • m -projection : m -project the current q to P by minimizing D KL ( q (cid:107) p ) over p ∈ P .
Revised: • m -projection : m -project the current q to P by minimizing D KL ( q (cid:107) p ) over p ∈ P .

Original: e  and m projections derive their names from the manifold of exponential family distributions and the manifold of mixture distributions respectively.
Revised: e - and m -projections derive their names from the manifold of exponential family distributions and the manifold of mixture distributions respectively.

Original: The two manifolds have different notions of ﬂatness.
Revised: The two manifolds have different notions of ﬂatness.

Original: The em -algorithm is closely related to the Expectation-Maximization (EM) algorithm
Revised: The em -algorithm is closely related to the Expectation-Maximization (EM) algorithm

Original: If we deﬁne Q = { p ( x, z | µ ) , µ = E q ( z im | x obs ) [ s ( x obs , z im )] , ∀ q ( z | x ) } , where q ( z | x ) is an arbitrary imputation distribution, then the em -algorithm becomes the EM algorithm.
Revised: If we deﬁne Q = { p ( x, z | µ ) , µ = E q ( z im | x obs ) [ s ( x obs , z im )] , ∀ q ( z | x ) } , where q ( z | x ) is an arbitrary imputation distribution, then the em -algorithm becomes the EM algorithm.

Original: In the above, we gloss over the difference between a single observation and repeated observations.
Revised: In the above, we gloss over the difference between a single observation and repeated observations.

Revised: See Appendix for an in-depth explanation of information geometry of e- and m-projections.

Original: Moving beyond exponential family models .
Revised: Moving beyond exponential family models .

Original: The original em and EM algorithms are formulated within the framework of exponential family models (1).
Revised: The original em and EM algorithms are formulated within the framework of exponential family models (1).

Original: Of course, the notion of e - and m -projections is not restricted to the exponential family, and only relies on the information-geometric notion of e - and m -ﬂatness of submanifolds that these projections take place.

Original: In fact, it is a manifestation of the so-called “generalized Pythagorean relation” [5] satisﬁed by the Kullback-Leibler divergence D

Original: KL ( q (cid:107) p ) .

Original: Let us elaborate below (see Figure 1).

Original: The Kullback-Leibler divergence D KL ( ·(cid:107)· ) can easily be shown to satisfy the following identity

Original: [Equation]

Original: (3) for any three probability distributions p, q, r .

Original: Now suppose we construct a one-parameter family q ( t ) , t ∈ [0 , 1] , of probability functions connecting p = q (1) to q = q (0) by q ( t ) =

Original: t p + (1 − t ) q, and another one-parameter famiy q ( s ) , parameterized by s ∈ [0 , 1] , of probability functions connecting r = q (1) to q = q (0) by

Original: [Equation]

Original: Here C ( s ) = (cid:0)(cid:82) r s q 1 − s (cid:1) − 1 is the normalization factor, with C (0)

Original: = 1 and

Original: [Equation]

Original: Each of these families form a curve segment on the manifold of probability distributions: (cid:101) pq := { q ( t ) } t ∈ [0 , 1] and (cid:101) qr :=

Original: { q ( s ) } s ∈ [0 , 1] are called m -geodesic and e -geodesic , respectively.

Original: They meet at the point q where s = t = 0 .

Original: The right-hand side of (3), denoted ∆ , becomes

Original: [Equation]

Original: s

Original: [Equation]

Original: Taking lim t → 0 , lim s → 0 in the above yields

Original: [Equation]

Original: because (cid:82) dqdt = ddt (cid:82) q = ddt 1 = 0 .

Original: Now suppose we denote θ as the coordinate system for the manifold of the parametric probability distributions, that is q = q θ .

Original: And the directional derivatives of (cid:101) pq and (cid:101)

Original: qr (at the intersecting point q ) are given by, respectively,

Original: [Equation]

Original: where V is the tangent vector for the m -geodesic curve (cid:101) pq at q and U the tangent vector for the e -geodesic curve (cid:101) qr at q . Hence [Equation] where
Revised: In discussing modern deep generative models below, we shall not limit ourselves to exponential family models, and we shall re-deﬁne the Q manifold.

Original: [Equation]

Original: is the Fisher information matrix, which is the Riemannian (Fisher-Rao) metric on the manifold of the probability functions.

Original: If the vectors V and U are chosen in such a way that makes ∆ = (cid:104) V, U (cid:105) = 0 , that is, they are “orthogonal” with respect to the Fisher-Rao metric (cid:104)· , ·(cid:105) , then (3) becomes the generalized Pythagorean relation

Original: [Equation]

Original: As each term in (4) is non-negative, we can consider D KL ( p (cid:107) q ) as a m -projection from p to a submanifold Q formed by the geodesic spray eminating from q ∈ Q , where q = argmin r D KL ( p (cid:107) r ) is the projection point of the ﬁxed p onto Q :

Original: [Equation]

Original: r ) .

Original: Likewise, we can consider D KL ( q (cid:107) r ) as an e -projection from r to a submanifold P formed by the geodesic spray eminating from q ∈ P , where q = argmin p D KL ( p (cid:107) r ) is the projection point of the ﬁxed r onto P :

Original: [Equation]

Original: The submanifold P is said to be m -ﬂat and it is made up of all m -geodesics of which (cid:101) pq is an instance, while the submanifold Q is said to be e -ﬂat

Original: and it is made up all e -geodesics of which (cid:101) qr is an instance.

Original: e -projection of point r to P (or m -projection of point p to Q) is unique when P is m -ﬂat (or Q is e -ﬂat, respectively).

Original: See Figure 2 for an illustration.

Original: Because of this general interpretation of the e - or m -projection, in discussing modern deep generative models below, we shall not limit ourselves to exponential family models, and we shall re-deﬁne the

Original: Q manifold.

Original: We shall focus on the key insight of joint minimization (2) and alternating projections.
Revised: We shall focus on the key insight of joint minimization (2) and alternating projections.

Original: In what follows, we shall connect this insight to the recent understanding of VAE [12].
Revised: In what follows, we shall connect this insight to the recent understanding of VAE [12].

Original:  VAE as alternating projections 3.1 Generator model
Revised:  VAE as alternating projections 3.1 Generator model

Original: The generator model [10, 17] is a non-linear generalization of factor analysis:
Revised: The generator model [10, 17] is a non-linear generalization of factor analysis:

Original: [Equation]
Revised: [Equation]

Original: where x is an image, or a sentence, or in general a high-dimensional example, z is a d -dimensional latent vector.
Revised: where x is an image, or a sentence, or in general a high-dimensional example, z is a d -dimensional latent vector.

Original: I d is the d -dimensional identity matrix.
Revised: I d is the d -dimensional identity matrix.

Original: g θ ( z ) is a top-down deep network, where θ consists of all the weight and bias terms.
Revised: g θ ( z ) is a top-down deep network, where θ consists of all the weight and bias terms.

Original: (cid:15) is usually assumed to be Gaussian white noise with meanand variance σ 2 .
Revised: (cid:15) is usually assumed to be Gaussian white noise with meanand variance σ 2 .

Original: Thus p θ ( x | z ) is N( g θ ( z ) , σ 2 I D ) , where D is the dimensionality of x .
Revised: Thus p θ ( x | z ) is N( g θ ( z ) , σ 2 I D ) , where D is the dimensionality of x .

Original: The generator model deﬁnes the joint probability model or the complete-data model p θ ( x, z )
Revised: The generator model deﬁnes the joint probability model or the complete-data model p θ ( x, z )

Original: =
Revised: =

Original: p ( z ) p θ ( x | z ) , where p ( z ) is the known prior model.
Revised: p ( z ) p θ ( x | z ) , where p ( z ) is the known prior model.

Original: We deﬁne P = { p θ ( x, z ) , ∀ θ } to be the model distribution manifold .
Revised: We deﬁne P = { p θ ( x, z ) , ∀ θ } to be the model distribution manifold .

Original: The marginal density or the observed-data model is p θ ( x ) =
Revised: The marginal density or the observed-data model is p θ ( x ) =

Original: (cid:82) p θ ( x, z ) dz , which is analytically implicit due to the intractable integral.
Revised: (cid:82) p θ ( x, z ) dz , which is analytically implicit due to the intractable integral.

Original: Let p data ( x ) be the data distribution that generates the observed x .
Revised: Let p data ( x ) be the data distribution that generates the observed x .

Original: The learning of p θ ( x ) can be based on min θ D KL ( p data ( x ) (cid:107) p θ ( x )) .
Revised: The learning of p θ ( x ) can be based on min θ D KL ( p data ( x ) (cid:107) p θ ( x )) .

Original: If we observe training examples { x i , i = 1 , ..., n } ∼ p data ( x ) , the above minimization can be approximated by maximizing the log-likelihood
Revised: If we observe training examples { x i , i = 1 , ..., n } ∼ p data ( x ) , the above minimization can be approximated by maximizing the log-likelihood

Original: [Equation]
Revised: [Equation]

Original: (6) which leads to maximum likelihood estimate (MLE).
Revised: which leads to maximum likelihood estimate (MLE).

Original: In this paper, we assume the sample size n is large enough so that the sample average over { x i , i = 1 , ..., n } is essentially the same as expectation with respect to p data .
Revised: In this paper, we assume the sample size n is large enough so that the sample average over { x i , i = 1 , ..., n } is essentially the same as expectation with respect to p data .

Original: 3.2 EM
Revised: 3.2 EM

Original: The EM algorithm can be understood by perturbing the KL-divergence for MLE.
Revised: The EM algorithm can be understood by perturbing the KL-divergence for MLE.

Original: Deﬁne D ( θ ) =
Revised: Deﬁne D ( θ ) =

Original: D
Revised: D

Original: KL ( p data ( x ) (cid:107) p θ ( x )) .
Revised: KL ( p data ( x ) (cid:107) p θ ( x )) .

Original: Then min θ D ( θ ) is equivalent to max θ L ( θ ) .
Revised: Then min θ D ( θ ) is equivalent to max θ L ( θ ) .

Original: Let θ t be the estimate at iteration t of EM.
Revised: Let θ t be the estimate at iteration t of EM.

Original: Let us consider the following perturbation of D ( θ ) ,
Revised: Let us consider the following perturbation of D ( θ ) ,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne p data ,θ t ( x, z )
Revised: where we deﬁne p data ,θ t ( x, z )

Original: = p data ( x ) p θ t ( z | x ) as the complete-data distribution at iteration t .
Revised: = p data ( x ) p θ t ( z | x ) as the complete-data distribution at iteration t .

Original: S ( θ ) is a surrogate for D ( θ ) at iteration t , and S ( θ ) is simpler than D ( θ ) because S ( θ ) is based on the joint distributions of ( x, z ) instead of the implicit marginal distribution of x as in D ( θ ) .
Revised: S ( θ ) is a surrogate for D ( θ ) at iteration t , and S ( θ ) is simpler than D ( θ ) because S ( θ ) is based on the joint distributions of ( x, z ) instead of the implicit marginal distribution of x as in D ( θ ) .

Original: The perturbation term D KL ( p θ t ( z | x ) (cid:107) p θ ( z | x )) , as a function of θ , achieves its minimum 0 at θ = θ t .
Revised: The perturbation term D KL ( p θ t ( z | x ) (cid:107) p θ ( z | x )) , as a function of θ , achieves its minimum 0 at θ = θ t .

Original: Being a minimum, its derivative at θ = θ t is zero.
Revised: Being a minimum, its derivative at θ = θ t is zero.

Original: Thus S ( θ ) ≥ D ( θ ) , i.e., S majorizes D , and they touch each other at θ t , where they share the same gradient at θ t , i.e., D (cid:48)
Revised: Thus S ( θ ) ≥ D ( θ ) , i.e., S majorizes D , and they touch each other at θ t , where they share the same gradient at θ t , i.e., D (cid:48)

Original: ( θ t ) =
Revised: ( θ t ) =

Original: S (cid:48) ( θ t ) .
Revised: S (cid:48) ( θ t ) .

Original: The
Revised: The

Original: EM algorithm ﬁnds θ t +1 = arg min θ S ( θ ) .
Revised: EM algorithm ﬁnds θ t +1 = arg min θ S ( θ ) .

Original: Since S ( θ t +1 ) ≤ S ( θ t ) , and S majorizes D , we have
Revised: Since S ( θ t +1 ) ≤ S ( θ t ) , and S majorizes D , we have

Original: D ( θ t +1 ) ≤ D ( θ t ) , which is the monotonicity of EM.
Revised: D ( θ t +1 ) ≤ D ( θ t ) , which is the monotonicity of EM.

Original: The above also underlies the gradient descent algorithm, in particular, the alternating back-propagation algorithm of [11], where D (cid:48) ( θ t ) is computed by S (cid:48)
Revised: The above also underlies the gradient descent algorithm, in particular, the alternating back-propagation algorithm of [11], where D (cid:48) ( θ t ) is computed by S (cid:48)

Original: ( θ t ) .
Revised: ( θ t ) .

Original: 3.3 EM as alternating projections
Revised: 3.3 EM as alternating projections

Original: Deﬁne q ( x, z ) =
Revised: Deﬁne q ( x, z ) =

Original: p data ( x ) q ( z | x ) , and re-deﬁne Q = { q ( x, z ) =
Revised: p data ( x ) q ( z | x ) , and re-deﬁne Q = { q ( x, z ) =

Original: p data ( x ) q ( z | x ) .
Revised: p data ( x ) q ( z | x ) .

Original: ∀ q
Revised: ∀ q

Original: ( z | x ) } , where p data ( x ) is ﬁxed, and q ( z | x ) can be any conditional distribution of z given x .
Revised: ( z | x ) } , where p data ( x ) is ﬁxed, and q ( z | x ) can be any conditional distribution of z given x .

Original: We may call q ( z | x ) the inference distribution or imputation distribution that imputes the latent z based on x . q ( x, z ) is the complete-data distribution .
Revised: We may call q ( z | x ) the inference distribution or imputation distribution that imputes the latent z based on x . q ( x, z ) is the complete-data distribution .

Original: Q is the data distribution manifold .
Revised: Q is the data distribution manifold .

Original: The EM algorithm alternates the following projections:
Revised: The EM algorithm alternates the following projections:

Original: • expectation-projection : Given the current p , min q D KL ( q (cid:107) p ) over q ∈ Q .
Revised: • expectation-projection : Given the current p , min q D KL ( q (cid:107) p ) over q ∈ Q .

Original: • m -projection :
Revised: • m -projection :

Original: Given the current q , min p D KL ( q (cid:107) p ) over p ∈ P .
Revised: Given the current q , min p D KL ( q (cid:107) p ) over p ∈ P .

Original: The m -projection remains the same as in the em -projections.
Revised: The m -projection remains the same as in the em -projections.

Original: The expectation-projection underlies the E-step.
Revised: The expectation-projection underlies the E-step.

Original: In the expectation-projection at iteration t with θ t , we can write
Revised: In the expectation-projection at iteration t with θ t , we can write

Original: [Equation]
Revised: [Equation]

Original: where the conditional KL divergence is deﬁned as
Revised: where the conditional KL divergence is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: with the outer expectation with respect to p data .
Revised: with the outer expectation with respect to p data .

Original: Clearly the minimization in expectation-projection is achieved at q ( z | x ) =
Revised: Clearly the minimization in expectation-projection is achieved at q ( z | x ) =

Original: p θ t ( z | x ) .
Revised: p θ t ( z | x ) .

Original: The minimization of the m -projection is to minimize S ( θ ) above.
Revised: The minimization of the m -projection is to minimize S ( θ ) above.

Original: 3.4 VAE
Revised: 3.4 VAE

Original: The generator model is commonly trained by variational auto-encoder (VAE) [17].
Revised: The generator model is commonly trained by variational auto-encoder (VAE) [17].

Original: VAE assumes an inference model q φ ( z | x ) as an approximation to the true posterior distribution p θ ( z | x ) .
Revised: VAE assumes an inference model q φ ( z | x ) as an approximation to the true posterior distribution p θ ( z | x ) .

Original: In
Revised: In

Original: VAE, q φ ( z | x ) is N( µ φ ( x ) , V φ ( x )) , where the mean vector µ φ and the diagonal variance-covariance matrix V φ are parametrized by bottom-up inference networks with a new set of parameters φ .
Revised: VAE, q φ ( z | x ) is N( µ φ ( x ) , V φ ( x )) , where the mean vector µ φ and the diagonal variance-covariance matrix V φ are parametrized by bottom-up inference networks with a new set of parameters φ .

Original: For z ∼ q φ ( z | x ) , we can write z = µ φ ( x ) + V φ ( x ) 1 / 2 w , where w ∼ N(0 , I d ) is Gaussian white noise.
Revised: For z ∼ q φ ( z | x ) , we can write z = µ φ ( x ) + V φ ( x ) 1 / 2 w , where w ∼ N(0 , I d ) is Gaussian white noise.

Original: Thus expectation with respect to z ∼ q φ ( z | x ) can be written as expectation with respect to w .
Revised: Thus expectation with respect to z ∼ q φ ( z | x ) can be written as expectation with respect to w .

Original: This reparameterization trick [17] helps reduce the variance in Monte Carlo integration.
Revised: This reparameterization trick [17] helps reduce the variance in Monte Carlo integration.

Original: q φ ( z | x ) is the learned inferential computation that approximately samples from p
Revised: q φ ( z | x ) is the learned inferential computation that approximately samples from p

Original: θ ( z | x ) .
Revised: θ ( z | x ) .

Original: Different from traditional variational inference [16, 20, 6], the parameters φ in q φ ( z | x ) are shared by all the training examples x .
Revised: Different from traditional variational inference [16, 20, 6], the parameters φ in q φ ( z | x ) are shared by all the training examples x .

Original: φ and θ can be trained together by jointly maximizing a lower bound of the log-likelihood.
Revised: φ and θ can be trained together by jointly maximizing a lower bound of the log-likelihood.

Original: 3.5 VAE as alternating projections
Revised: 3.5 VAE as alternating projections

Original: Now we re-deﬁne q data ,φ ( x, z ) =
Revised: Now we re-deﬁne q data ,φ ( x, z ) =

Original: p data ( x ) q φ ( z | x ) , and re-deﬁne Q = { q data ,φ ( x, z ) =
Revised: p data ( x ) q φ ( z | x ) , and re-deﬁne Q = { q data ,φ ( x, z ) =

Original: p data ( x ) q φ ( z | x ) , ∀ φ } .
Revised: p data ( x ) q φ ( z | x ) , ∀ φ } .

Original: We can formulate VAE as joint minimization min p ∈ P,q ∈ Q D KL ( q (cid:107) p ) , where
Revised: We can formulate VAE as joint minimization min p ∈ P,q ∈ Q D KL ( q (cid:107) p ) , where

Original: D KL ( q data ,φ ( x, z ) (cid:107) p θ ( x, z )) = D KL ( p data ( x ) (cid:107) p θ ( x ))
Revised: [Equation]

Original: + D KL ( q φ ( z | x ) (cid:107) p θ ( z | x )) , (9)
Revised: [Equation]

Original: On the right hand side, D ( θ ) =
Revised: On the right hand side, D ( θ ) =

Original: D KL ( p data ( x ) (cid:107) p θ ( x )) leads to the log-likelihood, and S ( θ, φ ) =
Revised: D KL ( p data ( x ) (cid:107) p θ ( x )) leads to the log-likelihood, and S ( θ, φ ) =

Original: D
Revised: D

Original: KL ( q data ,φ ( x, z ) (cid:107) p θ ( x, z ))
Revised: KL ( q data ,φ ( x, z ) (cid:107) p θ ( x, z ))

Original: =
Revised: =

Original: D ( θ ) + D KL ( q φ ( z | x ) (cid:107) p θ ( z | x )) is a perturbation of D ( θ ) .
Revised: D ( θ ) + D KL ( q φ ( z | x ) (cid:107) p θ ( z | x )) is a perturbation of D ( θ ) .

Original: Minimizing S ( θ, φ ) is equivalent to maximizing the evidence lower bound of the log-likelihood.
Revised: Minimizing S ( θ, φ ) is equivalent to maximizing the evidence lower bound of the log-likelihood.

Original: The minimization can be accomplished by alternating projections:
Revised: The minimization can be accomplished by alternating projections:

Original: • encoder-projection :
Revised: • encoder-projection :

Original: Given the current p , min q D KL ( q (cid:107) p ) over q ∈ Q .
Revised: Given the current p , min q D KL ( q (cid:107) p ) over q ∈ Q .

Original: • m -projection :
Revised: • m -projection :

Original: Given the current q , min p D KL ( q (cid:107) p ) over p ∈ P .
Revised: Given the current q , min p D KL ( q (cid:107) p ) over p ∈ P .

Original: Again the m -projection remains the same as in the em -algorithm.
Revised: Again the m -projection remains the same as in the em -algorithm.

Original: The encoder-projection in VAE is the same as the expectation-projection in EM, except that it is minimization over the inference model or the encoder model q φ ( z | x ) , instead of arbitrary q ( z | x ) as in EM.
Revised: The encoder-projection in VAE is the same as the expectation-projection in EM, except that it is minimization over the inference model or the encoder model q φ ( z | x ) , instead of arbitrary q ( z | x ) as in EM.

Original: The optimization is variational inference optimization [16, 20, 6], and the optimized q φ ( z | x ) may have a gap from the true posterior p θ ( z | x ) .
Revised: The optimization is variational inference optimization [16, 20, 6], and the optimized q φ ( z | x ) may have a gap from the true posterior p θ ( z | x ) .

Original: Figure 4 illustrates the alternating projections.
Revised: Figure 2 illustrates the alternating projections.

Original: In practice, the alternating projections can be implemented by alternating gradient descent.
Revised: In practice, the alternating projections can be implemented by alternating gradient descent.

Original: The wake-sleep algorithm [15, 14] is similar to VAE, except that it updates φ by min q D KL ( P (cid:107) Q ) based on sleep data generated from P .
Revised: The wake-sleep algorithm [15, 14] is similar to VAE, except that it updates φ by min q D KL ( P (cid:107) Q ) based on sleep data generated from P .

Original: Compared to encoder-projection, the order in KL divergence is reversed.
Revised: Compared to encoder-projection, the order in KL divergence is reversed.

Original: As is well known, reversing the order of KL divergence in minimization leads to different behaviors.
Revised: As is well known, reversing the order of KL divergence in minimization leads to different behaviors.

Original: 3.6 Short-run inference
Revised: 3.6 Short-run inference

Original: Between EM and VAE, we can strike a middle ground, where we use the short-run MCMC for approximate inference [22].
Revised: Between EM and VAE, we can strike a middle ground, where we use the short-run MCMC for approximate inference [22].

Original: Speciﬁcally, for the true posterior p θ ( z | x ) , we can use Langevin dynamics for short-run MCMC:
Revised: Speciﬁcally, for the true posterior p θ ( z | x ) , we can use Langevin dynamics for short-run MCMC:

Original: [Equation]
Revised: [Equation]

Original: where we initialize the dynamics from the ﬁxed prior distribution of z , i.e., p ( z ) ∼ N(0 , I d ) , and e k ∼ N(0 , I d ) is the Gaussian white noise.
Revised: where we initialize the dynamics from the ﬁxed prior distribution of z , i.e., p ( z ) ∼ N(0 , I d ) , and e k ∼ N(0 , I d ) is the Gaussian white noise.

Original: The dynamics runs a ﬁxed number of K steps with step size s .
Revised: The dynamics runs a ﬁxed number of K steps with step size s .

Original: We use q θ ( z | x ) to represent the distribution of z K .
Revised: We use q θ ( z | x ) to represent the distribution of z K .

Original: Then the learning with short-run inference is based on
Revised: Then the learning with short-run inference is based on

Original: [Equation]
Revised: [Equation]

Original: With θ t ﬁxed at iteration t , θ t +1 is obtained by gradient descent on S ( θ ) .
Revised: With θ t ﬁxed at iteration t , θ t +1 is obtained by gradient descent on S ( θ ) .

Original: The learning algorithm is a perturbation of maximum likelihood gradient ascent, where the perturbation is D
Revised: The learning algorithm is a perturbation of maximum likelihood gradient ascent, where the perturbation is D

Original: KL ( q θ t ( z | x ) (cid:107) p θ ( z | x )) .
Revised: KL ( q θ t ( z | x ) (cid:107) p θ ( z | x )) .

Original: As K → ∞ , D KL ( q θ ( z | x ) (cid:107) p θ ( z | x ))
Revised: As K → ∞ , D KL ( q θ ( z | x ) (cid:107) p θ ( z | x ))

Original: → 0 monotonically [7].
Revised: → 0 monotonically [7].

Original: Unlike VAE, q θ ( z | x ) does not involve a new set of variational parameters φ .
Revised: Unlike VAE, q θ ( z | x ) does not involve a new set of variational parameters φ .

Original: It is still based on the parameters θ of the generator model.
Revised: It is still based on the parameters θ of the generator model.

Original: However, we can optimize the small number of algorithmic parameters such as step size by minimizing D KL ( q θ t ( z | x ) (cid:107) p θ ( z | x ))
Revised: However, we can optimize the small number of algorithmic parameters such as step size by minimizing D KL ( q θ t ( z | x ) (cid:107) p θ ( z | x ))

Original:  Conclusion
Revised:  Conclusion

Original: This paper reviews the information geometric
Revised: This paper reviews the information geometric

Original: em -algorithm and the recent understanding of VAE.
Revised: em -algorithm and the recent understanding of VAE.

Original: While VAE does not share the framework of exponential family models with the em -algorithm, they share the same theoretical framework of joint minimization of KL divergence between two manifolds of distributions, as well as alternating projections (or alternating gradient descent) computation.
Revised: While VAE does not share the framework of exponential family models with the em -algorithm, they share the same theoretical framework of joint minimization of KL divergence between two manifolds of distributions, as well as alternating projections (or alternating gradient descent) computation.

Original: See [12] for an information geometric understanding of adversarial learning within the framework of energy-based models, and its integration with variational learning explained in this paper. 
Revised: See [12] for an information geometric understanding of adversarial learning within the framework of energy-based models, and its integration with variational learning explained in this paper. Acknowledgment

Revised: The work is supported by NSF DMS-2015577.

Revised: We thank the reviewers for their insightful comments and suggestions.

Revised: Appendix: information geometry of e- and m-projections

Revised: The notion of e - and m -projections is not restricted to the exponential family, and only relies on the information-geometric notion of e - and m -ﬂatness of submanifolds that these projections take place.

Revised: In fact, it is a manifestation of the so-called “generalized Pythagorean relation” [5] satisﬁed by the

Revised: Kullback-Leibler divergence D KL ( q (cid:107) p ) .

Revised: Let us elaborate below (see Figure 1).

Revised: The Kullback-Leibler divergence D KL ( ·(cid:107)· ) can easily be shown to satisfy the following identity

Revised: [Equation]

Revised: [Equation]

Revised: for any three probability distributions p, q, r .

Revised: Now suppose we construct a one-parameter family q ( t ) , t ∈ [0 , 1] , of probability functions connecting p = q (1) to q = q (0) by

Revised: [Equation]

Revised: ) q, and another one-parameter famiy q ( s ) , parameterized by s ∈ [0 , 1] , of probability functions connecting r = q (1) to q = q (0) by

Revised: [Equation]

Revised: Here C ( s ) = (cid:0)(cid:82) r s q 1 − s (cid:1) − 1 is the normalization factor, with C (0)

Revised: = 1 and

Revised: [Equation]

Revised: Each of these families form a curve segment on the manifold of probability distributions: (cid:101) pq := { q ( t ) } t ∈ [0 , 1] and (cid:101) qr :=

Revised: { q ( s ) } s ∈ [0 , 1] are called m -geodesic and e -geodesic , respectively.

Revised: They meet at the point q where s = t = 0 .

Revised: The right-hand side of (10), denoted ∆ , becomes ∆ ( t,s )

Revised: [Equation]

Revised: Taking lim t → 0 , lim s → 0 in the above yields

Revised: [Equation]

Revised: dt d log q ds , because (cid:82) dqdt = ddt (cid:82) q = ddt 1 = 0 .

Revised: Now suppose we denote θ as the coordinate system for the manifold of the parametric probability distributions, that is q = q θ .

Revised: And the directional derivatives of (cid:101) pq and (cid:101)

Revised: qr (at the intersecting point q ) are given by, respectively,

Revised: [Equation]

Revised: where V is the tangent vector for the m -geodesic curve (cid:101) pq at q and U the tangent vector for the e -geodesic curve (cid:101) qr at q .

Revised: Hence ∆ =

Revised: [Equation]

Revised: where

Revised: [Equation]

Revised: is the Fisher information matrix, which is the Riemannian (Fisher-Rao) metric on the manifold of the probability functions.

Revised: If the vectors V and U are chosen in such a way that makes ∆ = (cid:104) V, U (cid:105) = 0 , that is, they are “orthogonal” with respect to the Fisher-Rao metric (cid:104)· , ·(cid:105) , then (10) becomes the generalized Pythagorean relation

Revised: [Equation]

Revised: As each term in (11) is non-negative, we can consider D KL ( p (cid:107) q ) as a m -projection from p to a submanifold Q formed by the geodesic spray eminating from q ∈ Q , where q = argmin r D KL ( p (cid:107) r ) is the projection point of the ﬁxed p onto Q :

Revised: [Equation]

Revised: Likewise, we can consider D KL ( q (cid:107) r ) as an e -projection from r to a submanifold P formed by the geodesic spray eminating from q ∈ P , where q = argmin p D KL ( p (cid:107) r ) is the projection point of the ﬁxed r onto P :

Revised: [Equation]

Revised: The submanifold P is said to be m -ﬂat and it is made up of all m -geodesics of which (cid:101) pq is an instance, while the submanifold Q is said to be e -ﬂat

Revised: and it is made up all e -geodesics of which (cid:101) qr is an instance.

Revised: e -projection of point r to P (or m -projection of point p to Q) is unique when P is m -ﬂat (or Q is e -ﬂat, respectively).

Revised: See Figure 2 for an illustration.

Revised: the point q = argmin r D KL ( p (cid:107) r ) .