Recurrence Boosts Diversity! Revisiting Recurrent Latent Variable in Transformer-Based Variational A uto E ncoder for Diverse Text Generation (2022)

Abstract: Variational Auto-Encoder (VAE) has been widely adopted in text generation. Among many variants, recurrent VAE learns token-wise latent variables with each conditioned on the preceding ones, which captures sequential variability better in the era of RNN. However, it is unclear how to incorporate such recurrent dynamics into the recently dominant Transformer due to its parallelism. In this work, we propose TRACE, a Transformer-based recurrent VAE structure. TRACE imposes recurrence on segment-wise latent variables with arbitrarily separated text segments and constructs the posterior distribution with residual parameterization. Besides, we design an acceleration method by approximating idempotent matrices, which allows parallelism while maintaining the conditional dependence of latent variables. We demonstrate that TRACE could deduce a non-zero lower bound of the KL term and enhance the entanglement of each segment and preceding latent variables, providing a theoretical guarantee of generation diversity. Experiments on two unconditional and one conditional generation task show that TRACE achieves significantly improved diversity while maintaining satisfactory generation quality.

## introduction variational auto-encoder (vae) @xcite has thrived in various text generation tasks due to its ability to learn flexible representations, such as machine translation @xcite and the generation of dialogue @xcite , story @xcite and poetry @xcite . to further improve expressive power, diverse vae variants have been proposed. for example, iwae @xcite optimizes a tighter lower bound; ladder vae @xcite learns hierarchical latent representations and vmf-vae @xcite replaces gaussian distributions with von mises-fisher distributions. among all variants, temporal vae @xcite was prevalent in the era of rnn, which captures temporal variability by introducing the dependency of a series of latent variables with each associated with one time step. such a vae variant has succeeded in kinds of sequence modeling tasks, e.g., dialog generation @xcite , audio generation @xcite , and time series prediction @xcite . temporal vae can be categorized into three paradigms according to the dependency of prior distributions at each time step: a) independent normal distributions (abbr. ind) @xcite ), b) context-conditioned gaussian distributions (abbr. cgd) @xcite which are conditioned on preceding text, and c) recurrent gaussian distributions (abbr. rgd), i.e., recurrent vae @xcite , which are conditioned on preceding both text and latent variables foot_0 . both ind and cgd ignore the interaction of latent variables, limiting their expressive ability. in comparison, by introducing the dependency of latent variables, rgd could better model the sequential variability and thus greatly improve generation diversity while maintaining satisfactory quality. we provide the theoretical proof of such an advantage in sec. 4.3. these paradigms can be easily implemented with rnn benefiting from rnn's natural recurrent structure. stepping into the age of transformer @xcite , it is promising to adapt temporal vae to this popular architecture. ind and cgd paradigms are naturally compatible with transformer because their latent variables at each time step are independent which could be simply combined with the parallel computation of transformer self-attention via causal and noncausal masks @xcite . however, there are no off-the-shelf solutions to incorporate rgd into transformer-based vaes, since recurrence would be a natural obstacle to parallelism (recurrent latent variables need to be sequentially sampled), which limits the capacity of this potential vae paradigm. could we equip transformer with such recurrent dynamics for better diversity while keeping the training parallelism? to answer this question, we propose trace 2 , a novel transformerbased recurrent vae structure. trace imposes recurrence on segment-wise (instead of token-wise) latent variables with arbitrary segmentation, e.g., sentences or segments with specified length. besides, we construct the posterior distribution using residual parameterization and layer normalization, which could deduce a non-zero lower bound of the kl loss to alleviate kl vanishing @xcite . moreover, to accelerate training, we design a method to recover the parallelism in transformer by approximating idempotent parameter matrices for the latent space, leading to improved diversity, satisfactory quality, and faster training. in summary, our contributions are as follows: we are the first to ( i ) incorporate recurrent vae into transformer with recurrence on segment-wise latent variables which allows a flexible trade-off of diversity and quality; (ii) propose a method to recover parallelism and accelerate training with comparable performance; (iii) mathematically demonstrate that our model has a deducted non-zero lower bound to mitigate kl vanishing, and a theoretical interpretation of diversity improvement. (iv) we validate the effectiveness of our model on two unconditional and one conditional generation tasks. ## related work vae has shown great effectiveness in a wide range of nlg tasks, such as storytelling @xcite , dialogue generation @xcite and poetry composition @xcite . to further improve the expressive ability of vae, researchers propose various variants, e.g., vmf-vae @xcite that replaces the latent distribution with von mises-fisher distribution, ml-vae @xcite that learns multi-level latent variables, and bn-vae @xcite ) that utilizes batch normalization to get a non-zero kl lower bound. among all variants, temporal vae is the most prevalent one in the era of rnn, which introduces latent variables at each timestep and could natu-2 trace: transformer recurrent autoencoder rally fit with the recurrent structure of rnn. existing temporal vae fall into three paradigms according to the parameterization and dependence of the latent variables' prior distributions, namely ind, cgd, and rgd, as mentioned in sec. 1. for example, twr-vae @xcite utilizes a timestep-wise regularisation through independent latent variables with ind. vad @xcite incorporates cgd into latent variables and augments the posterior distribution with a backward rnn. recurrent vae @xcite learns token-wise latent variables with each sequentially conditioned on preceding ones as well as the context (i.e., rgd). by modeling the trajectory of both observed text sequences and latent space, recurrent vae could capture the sequential variability better @xcite . besides, we will show that such recurrent dynamics could theoretically reinforce the dependence on the stochastic and generalized latent space, thus boosting generation diversity by a large margin. recently, with the flourishing of the powerful transformer architecture, researchers have devoted to combining it with vae for text modeling and generation @xcite @xcite . vaes could promote generation diversity with satisfactory quality, benefiting from the intrinsic randomness in latent space. therefore, vae-based transformers are essential for various tasks demanding creativity, such as advertising text generation @xcite . two of the temporal vae paradigms, ind and cgd, can be easily adapted into transformer. for instance, svt @xcite applies cgdbased vae to dialogue generation. nonetheless, the integration of recurrent vae is still an open challenge due to the conflict in the parallelism in transformer and recurrent dependence of recurrent vae. to fully exploit the expressive power of recurrence, we revisit recurrent vae in transformer and propose trace which possesses advantages of both generation diversity and training parallelism. ## method to incorporate recurrent vae (rgd) into transformer, we propose trace that learns recurrent segment-wise latent variables and design an acceleration method to make full use of the advantage of parallelism in transformer. we present the adaption of recurrent vae to transformer and residual parameterization in sec. 4.1, demonstrate the parallel training method in sec. 4.2, and provide a theoretical interpretation of trace's effectiveness for boosting diversity in sec. 4.3. ## conclusion in this paper, we revisit the recurrent vae framework prevalent in the era of rnn, and propose a novel transformer-based recurrent vae structure trace. trace learns a series of segment-wise latent variables conditioned on the preceding ones. we establish the latent distributions with novel residual parameterization. to accelerate training, we design an approximate algorithm of learning latent variables to fit with the transformer framework. experimental shows that trace achieves significant improvement in generation diversity based on the tight relationship between text hidden states and the latent space. in the future, we will further explore the potential of trace in other larger pretrained models like gpt-3.