Effective Estimation of Deep Generative Language Models (2020)

Abstract: Advances in variational inference enable parameterisation of probabilistic models by deep neural networks. This combines the statistical transparency of the probabilistic modelling framework with the representational power of deep learning. Yet, due to a problem known as posterior collapse, it is difficult to estimate such models in the context of language modelling effectively. We concentrate on one such model, the variational auto-encoder, which we argue is an important building block in hierarchical probabilistic models of language. This paper contributes a sober view of the problem, a survey of techniques to address it, novel techniques, and extensions to the model. To establish a ranking of techniques, we perform a systematic comparison using Bayesian optimisation and find that many techniques perform reasonably similar, given enough resources. Still, a favourite can be named based on convenience. We also make several empirical observations and recommendations of best practices that should help researchers interested in this exciting field.

## introduction deep generative models (dgms) are probabilistic latent variable models parameterised by neural networks (nns). specifically, dgms optimised with amortised variational inference and reparameterised gradient estimates @xcite , better known as variational auto-encoders (vaes), have spurred much interest in various domains, including computer vision and natural language processing (nlp). in nlp, vaes have been developed for word representation @xcite , morphological analysis @xcite , syntactic and work done while the first author was at the university of amsterdam. code is available at https://github.com/ tom-pelsmaeker/deep-generative-lm semantic parsing @xcite , document modelling @xcite , summarisation @xcite , machine translation @xcite @xcite , language and vision @xcite , dialogue modelling @xcite , speech modelling @xcite , and, of course, language modelling @xcite . one problem remains common to the majority of these models, vaes often learn to ignore the latent variables. we investigate this problem, dubbed posterior collapse, in the context of language models (lms). in a deep generative lm @xcite , sentences are generated conditioned on samples from a continuous latent space, an idea with various practical applications. for example, one can constrain this latent space to promote generalisations that are in line with linguistic knowledge and intuition @xcite . this also allows for greater flexibility in how the model is used, for example, to generate sentences that live-in latent space-in a neighbourhood of a given observation @xcite . despite this potential, vaes that employ strong generators (e.g. recurrent nns) tend to ignore the latent variable. figure 1 illustrates this point: neighbourhood in latent space does not correlate to patterns in data space, and the model behaves just like a standard lm. recently, many techniques have been proposed to address this problem ( §3 and §7) and they range from modifications to the objective to changes to the actual model. some of these techniques have only been tested under different conditions and under different evaluation criteria, and some of them have only been tested outside nlp. this paper contributes: (1) a novel strategy based on constrained optimisation towards a pre-specified upper-bound on mutual information; (2) multimodal priors that by design promote increased mutual information between data and latent code; last and, arguably most importantly, (3) a systematic comparisonin terms of resources dedicated to hyperparameter search and sensitivity to initial conditions-of strategies to counter posterior collapse, including some never tested for language models (e.g. in-fovae, lagvae, soft free-bits, and multimodal priors). ## density estimation for text density estimation for written text has a long history @xcite , but in this work we concentrate on neural network models @xcite , in particular, autoregressive ones @xcite . following common practice, we model sentences independently, each a sequence @xmath0, . . . , x n of @xmath1. ## posterior collapse in vi, we make inferences using an approximation q(z|x, λ) to the true posterior p(z|x, θ) and choose λ as to minimise the kl divergence e x [kl(q(z|x, λ)||p(z|x, θ))]. the same principle yields a lower-bound on log-likelihood used to estimate θ jointly with λ, thus making the true posterior p(z|x, θ) a moving target. if the estimated conditional p (x|z, θ) can be made independent of z, which in our case means relying exclusively on @xmath2, the true posterior will be independent of the data and equal to the prior. 3 based on such observation, @xcite argue that information that can be modelled by the generator without using latent variables will be modelled that way-precisely because when no information is encoded in the latent variable the true posterior equals the prior and it is then trivial to reduce e x [kl(q(z|x, λ)||p(z|x, θ))] to 0. this is typically diagnosed by noting that after training kl(q(z|x, λ)||p(z)) → 0 for most x: we say that the true posterior collapses to the prior. alemi et al. (2018) show that the rate, @xmath3, λ)||p(z))], is an upperbound to i(x; z|λ), the mutual information (mi) between x and z. thus, if kl(q(z|x, λ)||p(z)) is 3 this follows trivially from the definition of posterior: p(z|x) = p(z)p (x|z) p (x) x⊥@xmath4 a generator that makes no markov assumptions, such as a recurrent lm, can potentially achieve x i ⊥ z | @xmath5, θ, and indeed many have noticed that vaes whose observation models are parameterised by such strong generators (or strong decoders) tend to ignore the latent representation @xcite @xcite . for this reason, a strategy to prevent posterior collapse is to weaken the decoder @xcite @xcite . in this work, we are interested in employing strong generators, thus we do not investigate weaker decoders. other strategies involve changes to the optimisation procedure and manipulations to the objective that target local optima of the elbo with non-negligible mi. annealing @xcite propose "kl annealing", whereby the kl term in the elbo is incorporated into the objective in gradual steps. this way the optimiser can focus on reducing distortion early on in training, potentially by increasing mi. they also propose to drop words from @xmath6. we experiment with a slight modification of word dropout whereby we slowly vary the dropout rate from 1 → 0. in a sense, we "anneal" from a weak to a strong generator. targeting rates another idea is to target a prespecified rate @xcite . @xcite replace the kl term in the elbo with max(r, kl(q(z|x, λ)||p(z))), dubbed free bits (fb) because it allows encoding the first r nats of information "for free". as long as kl(q(z|x, λ)||p(z)) < r, this does not optimise a proper elbo (it misses the kl term), and the max introduces a discontinuity. @xcite propose soft free bits (sfb), that instead multiplies the kl term in the elbo with a weighing factor 0 < β ≤ 1 that is dynamically adjusted based on the target rate r: β is incremented (or reduced) by ω if @xmath7. note that this technique requires hyperparameters (i.e. γ, ε, ω) besides r to be tuned in order to determine how β is updated. ## minimum desired rate we propose minimum desired rate (mdr), a technique to attain elbo values at a pre-specified rate r that does not suffer from the gradient discontinuities of fb, and does not introduce the additional hyperparameters of sfb. the idea is to optimise the elbo subject to a minimum rate constraint r: ## expressive priors suppose we employ a multimodal prior p(z|θ), e.g. a mixture of gaussians, and suppose we employ a unimodal posterior approximation, e.g. the typical diagonal gaussian. this creates a mismatch between the prior and the posterior approximation families that makes it impossible for kl(q(z|x, λ)||p(z|θ)) to be precisely 0. for the aggregated posterior q(z|λ) to match the prior, the inference model would have to-on average-cover all of the prior's modes. since the inference network is deterministic, it can only do so as a function of the conditioning input x, thus increasing i(x; z|λ). admittedly, this conditioning might still only capture shallow features of x, and the generator may still choose to ignore the latent code, keeping i(x; z|θ) low, but the potential seems to justify an attempt. this view builds upon alemi et al. ( 2018 )'s information-theoretic view which suggests that the prior regularises the inference model capping i(x; z|λ). thus, we modify sen-vae to employ a more complex, ideally multimodal, parametric prior p(z|θ) and fit its parameters. mog our first option is a uniform mixture of gaussians (mog), i.e. p(z|θ vampprior motivated by the fact that, for a fixed posterior approximation, the prior that optimises the elbo equals e x [q(z|x, λ)], tomczak and welling (2018) propose the vampprior, a variational mixture of posteriors: ## experiments our goal is to identify which techniques are effective in training vaes for language modelling. our evaluation concentrates on intrinsic metrics: negative log-likelihood (nll), perplexity per token (ppl), rate (r), distortion (d), the number of active units (au; burda et al., 2015)) 5 and gap in the accuracy of next word prediction (given gold prefixes) when decoding from a posterior sample versus decoding from a prior sample (acc gap ). for vae models, nll (and thus ppl) can only be estimated. we use importance sampling (is) with our trained approximate posterior as importance distribution (we use @xmath8. we first report on experiments using the english penn treebank (ptb; @xcite . 6 rnnlm the baseline rnnlm generator is a building block for all of our senvaes, thus we validate its performance as a strong standalone generator. we highlight that it outperforms an external baseline that employs a comparable number of parameters @xcite and that this performance boost is mostly due to tying embeddings with the output layer. 7 appendix a.1 presents the complete architecture and a comparison. ## related work in nlp, posterior collapse was probably first noticed by @xcite , who addressed it via word dropout and kl scaling. further investigation revealed that in the presence of strong generators, the elbo itself becomes the culprit @xcite , as it lacks a preference regarding mi. posterior collapse has also been ascribed to approximate inference @xcite . beyond the techniques compared and developed in this work, other solutions have been proposed, including modifications to the generator @xcite @xcite , side losses based on weak generators @xcite , factorised likelihoods @xcite , cyclical annealing @xcite and changes to the elbo @xcite . exploiting a mismatch in correlation between the prior and the approximate posterior, and thus forcing a lower-bound on the rate, is the principle behind δ-vaes @xcite and hyperspherical vaes @xcite . the generative model of δ-vaes has one latent variable per step of the sequence, i.e. @xmath9, . . . , z |x| , making it quite different from that of the sen-vaes considered here. their mean-field inference model is a product of independent gaussians, one per step, but they construct a correlated gaussian prior by making the prior distribution over the next step depend linearly on the previous step, i.e. z i |z i-1 ∼ n (αz i-1 , σ) with hyperparameters α and σ. hyperspherical vaes work on the unit hypersphere with a uniform prior and a nonuniform vonmises-fisher posterior approximation @xcite . note that, though in this paper we focused on gaussian (and mixture of gaussians, e.g. mog and vampprior) priors, mdr is applicable for whatever choice of prescribed prior. whether its benefits stack with the effects due to different priors remains an empirical question. @xcite casts vae optimisation as a dual problem, and in that it is closely related to our mdr and the lagvae. geco targets minimisation of e x [kl(q(z|x, λ)||p(z))] under constraints on distortion, whereas lagvae targets either maximisation or minimisation of (bounds on) i(x; z|λ) under constraints on the infovae objective. contrary to mdr, geco focuses on latent space regularisation and offers no explicit mechanism to mitigate posterior collapse. recently @xcite proposed to combine fb, kl scaling, and pre-training of the inference network's encoder on an auto-encoding objective. their techniques are complementary to ours in so far as their main finding-the mutual benefits of annealing, pre-training, and lower-bounding kl-is perfectly compatible with ours (mdr and multimodal priors). ## discussion senvae is a deep generative model whose generative story is rather shallow, yet, due to its strong generator component, it is hard to make effective use of the extra knob it offers. in this paper, we have introduced and compared techniques for effective estimation of such a model. we show that many techniques in the literature perform reasonably similarly (i.e. fb, sfb, β-vae, infovae), though they may require a considerable hyperparameter search (e.g. sfb and infovae). amongst these, our proposed optimisation subject to a minimum rate constraint is simple enough to tune (as fb it only takes a pre-specified rate and unlike fb it does not suffer from gradient discontinuities), superior to annealing and word dropout, and require less resources than strategies based on multiple annealing schedules and/or aggressive optimisation of the inference model. other ways to lower-bound rate, such as by imposing a multimodal prior, though promising, still require a minimum desired rate. the typical rnnlm is built upon an exact factorisation of the joint distribution, thus a welltrained architecture is hard to improve upon in terms of log-likelihood of gold-standard data. our interest in latent variable models stems from the desire to obtain generative stories that are less opaque than that of an rnnlm, for example, in that they may expose knobs that we can use to control generation and a hierarchy of steps that may award a degree of interpretability to the model. the sen-vae is not that model, but it is a crucial building block in the pursue for hierarchical probabilistic models of language. we hope this work, i.e. the organised review it contributes and the techniques it introduces, will pave the way to deeper-in statistical hierarchy-generative models of language.