  This is a tutorial and survey paper on factor analysis, probabilistic
Principal Component Analysis (PCA), variational inference, and Variational
Autoencoder (VAE). These methods, which are tightly related, are dimensionality
reduction and generative models. They asssume that every data point is
generated from or caused by a low-dimensional latent factor. By learning the
parameters of distribution of latent space, the corresponding low-dimensional
factors are found for the sake of dimensionality reduction. For their
stochastic and generative behaviour, these models can also be used for
generation of new data points in the data space. In this paper, we first start
with variational inference where we derive the Evidence Lower Bound (ELBO) and
Expectation Maximization (EM) for learning the parameters. Then, we introduce
factor analysis, derive its joint and marginal distributions, and work out its
EM steps. Probabilistic PCA is then explained, as a special case of factor
analysis, and its closed-form solutions are derived. Finally, VAE is explained
where the encoder, decoder and sampling from the latent space are introduced.
Training VAE using both EM and backpropagation are explained.
