Learning VAE - LDA Models with Rounded Reparameterization Trick (2020)

Abstract: The introduction of VAE provides an efficient framework for the learning of generative models, including generative topic models. However, when the topic model is a Latent Dirichlet Allocation (LDA) model, a central technique of VAE, the reparameterization trick, fails to be applicable. This is because no reparameterization form of Dirichlet distributions is known to date that allows the use of the reparameterization trick. In this work, we propose a new method, which we call Rounded Reparameterization Trick (RRT), to reparameterize Dirichlet distributions for the learning of VAE-LDA models. This method, when applied to a VAE-LDA model, is shown experimentally to outperform the existing neural topic models on several benchmark datasets and on a synthetic dataset.

## introduction probabilistic generative models are widely used in topic modelling and have achieved great success in many applications @xcite @xcite . a landmark of topic models is latent dirichlet allocation (lda) @xcite , where a document is treated as a bag of words and each word is modelled via a generative process. more specifically, in this generative process, a topic distribution is first drawn from a dirichlet prior, then a topic is sampled from the topic distribution and a word is drawn subsequently from the word distribution corresponding to the drawn topic. since its introduction, lda has shown great power in a large varieties of natural language applications @xcite @xcite . however, the classical methods of learning lda, such as variational techniques and collapsed gibbs sampling, entails high computation complexity in posterior inference @xcite , which limits the ability of lda on modelling large corpus. variational autoencoder (vae) or autoencoding variational bayes (aevb) @xcite provides another choice of learning a generative model. under the vae framework, a generative model is specified by first drawing a latent vector z from a prior distribution and then transforming this vector through a neural network, called decoder, which subsequently generates the observation x. using a variational inference approach, vae couples the decoder network with another network, called encoder, responsible for computing the posterior distribution of the latent variable z for each observation x. a key technique of vae is its "reparameterization trick", in which sampling from the posterior is performed by sampling a noise variable from some distribution p( ) and then transforming to z using a differentiable function. this technique allows the model to be trained efficiently using back propagation. the vae framework significantly alleviates the computational burden of learning a generative model. therefore, researchers interested in topic modelling are naturally motivated to consider vae as an alternative approach to learn lda, exploiting the power and efficiency of deep learning neural networks. this is also the interest of this paper. however, the key limitation in the application of vae to dirichlet-based topic models is that the original reparameterization trick in vae is not applicable to dirichlet distributions. in this sense, vae cannot be directly used for learning any dirichlet-based topic models. to cope with this, the nvdm model @xcite discards the dirichlet assumption and build neural topic models based on gaussian prior. although such a gaussian-based topic model achieves a reasonably good performance on perplexity, the topic words they extracted appear to lack human-interpretability. additionally the use of gaussian prior significantly deviates from the desired dirichlet distribution and arguably has significant room for improvement. the adoption of the dirichlet prior plays a central role in topic modelling, since it nicely captures the intuition that a topic is sampled from a sparse topic distribution. due to the importance of the dirichlet assumption in topic modelling, prodlda @xcite attempts to apply vae to lda by constructing a laplace approximation to the dirichlet prior in the softmax basis. however, the laplace approximation is only used to estimate the prior parameters and prodlda has essentially a gaussian vae architecture where the kl divergence is on gaussian distributions. the work of @xcite argues that the laplace approximation in prodlda fails to capture the multimodality nature of dirichlet distributions. they then propose dirvae, in which an approximation of the inverse gamma cdf (knowles, 2015) is used to reparameterize gamma distributions. the dirichlet samples are then constructed by normalizing gamma random variables. however, the approximation of inverse gamma cdf is accurate only when the shape parameter of the gamma distribution is much less than 1 @xcite . this in turn limits the application scope of dirvae. in this work, we develop a technique, which we call the rounded reparameterization trick (rrt), to reparameterize dirichlet distributions. the use of rrt enables vae as an efficient method for learning lda, based on which we propose a new neural topic model, referred to as "rrt-vae". 1 experiments on several datasets show that rrt-vae outperforms nvdm, prodlda, and dirvae. the experimental results strongly demonstrate the applicability of rrt in topic modelling that utilizes vae. ## rounded reparameterization trick to tackle the limitation of the standard reparameterization trick, we propose a new reparameterization method, referred to as rounded reparameterization trick or rrt. given a real number ∆, we define a "∆rounding" function • ∆ as follows: for any real number a, in rrt, we draw an auxiliary variable θ from a "rounded" posterior distribution q θ| α ∆ , then θ is used to approximate θ ∼ q(θ|α). in (12), the parameter λ is a hyper parameter which will serve to adjust the strength of the gradient. note that when choosing a very small rounding precision ∆, we expect that the distribution q(•|α) of θ and the distribution q(•|α) are nearly identical. as a consequence, e q(θ|α) [j(θ, x)] and its replacement e q(θ|α) [j(θ, x)] are also very close to each other. thus such a replacement keeps the loss function very close to the original loss in (8). the gradient ∇ α a(α) can be expressed as a sum of two terms: the first term in sum is usually estimated through the score function estimator. but this is unnecessary in this case. to see this, note that ∇ α α ∆ = 0 almost everywhere. this implies that the first term is in fact 0 at every α for which the gradient exists. the next lemma then immediately follows. the fact that the score function estimator is not needed for estimating the gradient ∇ α a(α) allows rrt to enjoy a low variance and hence requires very few samples in monte-carlo estimation. using lemma 1, one can directly express the stochastic (monte carlo) estimate of the gradient at this end, we conclude that the loss function l obtained by replacing θ with θ is very close to the original loss function l, and a low-variance gradient estimator can be easily constructed from l. this completes the description of rrt. ## related work beyond topic modelling, another theme of research related to this work is the estimation of gradient in neural networks containing stochastic nodes or samplers. in this setting, one desires that the gradient signal is capable of back-propagating through the samplers. a classical method for this purpose is to construct a score function estimator, also known as the "log derivative trick" or reinforce @xcite . however, despite giving an unbiased estimate, the monte-carlo implementation of such an estimator typically suffers from a high variance, and thus relies on some additional variance-reduction techniques @xcite . reparameterization trick(kingma and welling, 2013), as mentioned above, may also be used to back-propagate gradients through samples and enjoys a low-variance advantage. unfortunately this technique is not applicable to many distributions such as gamma, beta and dirichlet distributions. various efforts have been spent on extending the applicability of reparameterization trick to a broader range. these works include, for example, g-rep @xcite , rsvi (naesseth et al., 2016) and implicit reparameterization gradients @xcite , etc. these methods usually involve complicated gradient derivations and are often difficult to implement in neural networks. ## experiments and results to quantitatively evaluate rrt-vae, we conduct experiments on synthetic datasets and five realworld datasets. our model is compared with several existing topic models: online lda @xcite , nvdm @xcite , prodlda (srivastava and sutton, 2017) and dirvae @xcite . in the experiments, we adopt three mlps with relu activations as the encoder of rrt-vae, where each hidden layer is set to 500 dimensions. we apply an exponential function on the outputs of the encoder, so that the outputs are positive values. the topic distribution vectors are sampled through rrt and then normalized before being passed to the decoder. for online lda, we use the standard implementation from scikit-learn @xcite . the encoder structures of nvdm, prodlda and dirvae are built according to @xcite , @xcite and @xcite , where in our experiments the dimension of each hidden layer is set to 500. on the real-world datasets, we adopt the prod decoder, since the standard decoder appears to extract many repetitive topic words (see appendix b.1). 2 2 as reported in @xcite , prodlda also appears to extract many repetitive words when using the on the synthetic datasets, we adopt the standard decoder, which is examined to be superior to the prod decoder on this learning task (see appendix a.1). ## concluding remarks in this paper, rounded reparameterization trick, or rrt, is shown as an effective and efficient reparameterization method for dirichlet distributions in the context of learning vae based lda models. in fact, the applicability of rrt can be generalized beyond dirichlet distributions. this is because any distribution can be reparameterized to an "rrt form" as long as a sampling algorithm exists for that distribution. thus it will be interesting to investigate the performance of rrt in other applications of vae beyond topic modelling. successes in these investigations will certainly extend the applicability of vae to much broader application domains and model families.