  Variational autoencoder (VAE) is a very successful generative model whose key
element is the so called amortized inference network, which can perform test
time inference using a single feed forward pass. Unfortunately, this comes at
the cost of degraded accuracy in posterior approximation, often underperforming
the instance-wise variational optimization. Although the latest semi-amortized
approaches mitigate the issue by performing a few variational optimization
updates starting from the VAE's amortized inference output, they inherently
suffer from computational overhead for inference at test time. In this paper,
we address the problem in a completely different way by considering a random
inference model, where we model the mean and variance functions of the
variational posterior as random Gaussian processes (GP). The motivation is that
the deviation of the VAE's amortized posterior distribution from the true
posterior can be regarded as random noise, which allows us to take into account
the uncertainty in posterior approximation in a principled manner. In
particular, our model can quantify the difficulty in posterior approximation by
a Gaussian variational density. Inference in our GP model is done by a single
feed forward pass through the network, significantly faster than semi-amortized
methods. We show that our approach attains higher test data likelihood than the
state-of-the-arts on several benchmark datasets.
