Addressing Posterior Collapse with Mutual Information for Improved Variational Neural Machine Translation (2020)

Abstract: This paper proposes a simple and effective approach to address the problem of posterior collapse in conditional variational autoencoders (CVAEs). It thus improves performance of machine translation models that use noisy or monolingual data, as well as in conventional settings. Extending Transformer and conditional VAEs, our proposed latent variable model measurably prevents posterior collapse by (1) using a modified evidence lower bound (ELBO) objective which promotes mutual information between the latent variable and the target, and (2) guiding the latent variable with an auxiliary bag-of-words prediction task. As a result, the proposed model yields improved translation quality compared to existing variational NMT models on WMT Ro↔En and De↔En. With latent variables being effectively utilized, our model demonstrates improved robustness over non-latent Transformer in handling uncertainty: exploiting noisy source-side monolingual data (up to +3.2 BLEU), and training with weakly aligned web-mined parallel data (up to +4.7 BLEU).

## introduction the conditional variational autoencoder (cvae; @xcite is a conditional generative model for structured prediction tasks like machine translation. this model, learned by variational bayesian methods @xcite , can capture global signal about the target in its latent variables. unfortunately, variational inference for text generation often yields models that ignore their latent variables @xcite , a phenomenon called posterior collapse. in this paper, we introduce a new loss function for cvaes that counteracts posterior collapse, motivated by our analysis of cvae's evidence lower bound objective (elbo). our analysis ( §2) reveals that optimizing elbo's second term not only brings the variational posterior approximation closer to the prior, but also decreases mutual information between latent variables and observed data. based on this insight, we modify cvae's elbo in two ways ( §3): (1) we explicitly add a principled mutual information term back into the training objective, and (2) we use a factorized decoder @xcite , which also predicts the target bagof-words as an auxiliary decoding distribution to regularize our latent variables. our objective is effective even without kullback-leibler term (kl) annealing @xcite , a strategy for iteratively altering elbo over the course of training to avoid posterior collapse. in applying our method to neural machine translation (nmt; @xcite , we find that we have measurably mitigated posterior collapse. the latent variables are not ignored, even in the presence of a powerful transformer decoder. by addressing this problem, the resulting nmt model has improved robustness and performance in low-resource scenarios. noisy data like those scraped from the internet @xcite present a challenge for nmt @xcite ; we are measurably more able to model this extrinsic uncertainty than the (non-latent) transformer @xcite or existing variational nmt with the cvae architecture @xcite . finally, we extend the model to semi-supervised learning @xcite to more effectively learn from monolingual data. in summary, our conditional text generation model overcomes posterior collapse by promoting mutual information. it can easily and successfully integrate noisy and monolingual data, and it does this without the cost of lower bleu score than non-latent nmt in typical settings. ## formalism and mathematical analysis here we review the standard framework for neural mt. next, we connect this to the conditional variational autoencoder, a model with latent random variables whose distributions are learned by blackbox variational bayesian inference. finally, we analyze the cvae's objective to explain why these models will ignore their latent variables ("posterior collapse"). ## an information-infused objective we modify our training objective to explicitly retain mutual information between the latent variable z and the observation (x, y). further, we use an auxiliary decoder that only uses the latent variable, not the encoder states. we combine it with the existing decoder as a mixture of softmaxes @xcite . the model is trained with amortized variational inference. when source-language monolingual text is available, we augment our modified cvae objective with a similarly modified (non-conditional) vae objective. the training and inference strategy is summarized in figure 1 . ## experiments and results here we present empirical results on the transformer architecture. we evaluate our model on four standard datasets and compare against three baselines. we use four measures to quantify posterior collapse, then examine translation quality (bleu score) in standard fully supervised settings, a semi-supervised setting, and a fully supervised setting with noisy source text. hyperparameters, regularization choices, and subword vocabulary information can be found in §5.3. the results show that we have effectively addressed posterior collapse: latent variables are no longer ignored despite the presence of a powerful decoder. as a result, we outperform both the standard transformer and the transformer-based variational nmt approach, when using noisy data or source-language monolingual data. ## analysis ablation study how do the different ingredients of our proposed approach contribute to preventing posterior collapse and improving translation quality? we explore two variants of the proposed model: 1) modified elbo only: only adding mutual information term to the training objective, while without gradients from l bow , 2) bow only: which is equivalent to dcvae combined with bow decoder. first, we perform the same collapse metrics evaluation as in analysis of outputs delving into model predictions helps us understand how our model outperforms the others. we examined erroneous 1-best predictions on the ro-en data. we provide salient examples of phenomena we identified in several examples support the fact that our model has more fluent and accurate translations than the baseline or vnmt. vnmt often struggles by introducing disfluent words, and both vnmt and transformer select justifiable but incorrect words. for instance, in our second example, the gender and animacy of the possessor are not specified in romanian. our model selects a more plausible pronoun for this context. analysis of latent variables finally, we probe whether different latent variables encode different information. we random sample 100 sentences from two test sets of distinct domains, mtnt (reddit comments) and wmt (news) with 50 sentences each. we plot the t-sne projection of their corresponding samples z k inferred from φ k , @xmath0, 2, 3, 4 respectively. figure 4 suggests that different latent variables learn to organize the data in different manners, but there was no clear signal that any of them exclusively specialize in encoding a domain label. we leave a thorough analysis of figure 4 : t-sne visualization of z k , @xmath1, 2, 3, 4 samples from 100 sentences from two datasets with distinct domains, mtnt (orchid) and wmt news (green). ## related work unlike most prior work in (conditional) text generation, we tackle posterior collapse without requiring an annealing schedule @xcite @xcite , a weakened decoder @xcite , or a restricted variational family @xcite . unlike @xcite , who also employ bag-ofwords as an nmt objective, our bow decoder only sees the latent variable z, not the encoder states. conversely, unlike @xcite , our generative decoder has access to both the latent variable and the encoder states; bag-of-words prediction is handled by separate parameters. vnmt @xcite applies cvae with gaussian priors to conditional text generation. vrnmt @xcite extends vnmt, mod-eling the translation process in greater granularity. both needed manually designed annealing schedules to increase kl loss and avoid posterior collapse. discrete latent variables have been applied to nmt @xcite @xcite , without variational inference or addressing posterior collapse. approaches to stop posterior collapse include aggressively trained inference networks @xcite , skip connections @xcite , and expressive priors @xcite . unlike our conditional approach, shah and barber (2018) jointly model the source and target text in a generative fashion. their em-based inference is more computationally expensive than our amortized variational inference. eikema and aziz (2019) also present a generative (joint) model relying on autoencoding; they condition the source text x on the latent variable z. finally, @xcite , like us, value mutual information between the data and the latent variable. while they motivate kl annealing using mutual information, we show that the annealing is unnecessary. ## conclusion we have presented a conditional generative model with latent variables whose distribution is learned with variation inference, then evaluated it in machine translation. our approach does not require an annealing schedule or a hamstrung decoder to avoid posterior collapse. instead, by providing a new analysis of the conditional vae objective to improve it in a principled way and incorporating an auxiliary decoding objective, we measurably prevented posterior collapse. as a result, our model has outperformed previous variational nmt models in terms of translation quality, and is comparable to non-latent transformer on standard wmt ro↔en and de↔en datasets. furthermore, the proposed method has improved robustness in dealing with uncertainty in data, including exploiting source-side monolingual data as well as training with noisy parallel data.