Empirical Prior for Text Autoencoders (2024)

Abstract: This paper explores the application of Variational Autoencoders (VAE) in text generation, focusing on overcoming challenges like posterior collapse and the limitations of simplistic prior distributions. We investigate a transition from VAE to text autoencoders (AE), which model a compact latent space and preserves the capability of the language model itself. Our method involves layer-wise latent vectors regularized by orthogonal constraints to encourage distinct semantic spaces. In particular, we estimate an empirical prior online from the learned latent vectors to support sampling during generation like VAE. Experimental results on standard benchmarks demonstrate that the autoencoders generate higher quality and more diverse text than the VAE-based Transformer baselines, offering an effective alternative for generative language modeling.

## introduction variational autoencoder (vae) offers an effective approach to train generative models with latent variables @xcite . by adopting the paradigm for training language models, the latent variables can help to capture the underlying causal structure of the generative process more effectively, and provide an interpretable representation of high-level features like topics or syntactic properties @xcite @xcite . moreover, representing sentences in a lowdimensional latent space facilitates manipulation and guided generation using interpretable vector operators. vaes have demonstrated their success in generating stylistic text @xcite , stories @xcite , and dialog @xcite . optimus @xcite , the pioneering large-scale pretrained vae for text, underscoring the advantages highlighted during the pre-training phase. while vaes possess promising theoretical advantages, they face challenges such as posterior collapse @xcite @xcite , successfully training these models necessitates the use of carefully @xcite @xcite @xcite , and the learned posterior often deviates significantly from the hypothetical prior, leading to undermining the quality of the generated text. the drawbacks of vae can be attributed to the simplistic prior and its restriction on the latent distribution induced by the kl divergence term @xcite (the upper part of figure 1 ). following the practice of vae in image generation, text vae adopts such a prior to represent plausible language. however, autoregressive language models themselves possess a strong ability to generate linguistically valid outputs, which weaken the role of the hypothetical prior @xcite @xcite . based on the above observations, we investigate the potential of ae in achieving what can be achieved by text vae. typically, vae degenerates to ae when removing the regularization of samples to the prior. the text ae precisely preserves the semantics of the input but lacks the capability of generative modeling, specifically the ability to sample novel latent vectors. to address this issue, we propose to estimate the prior based on the learned features of ae, i.e., empirical prior distribution. the whole model is illustrated in the lower half of figure 1 . specifically, we introduce multiple latent vectors for ae induced from different encoder layers. to encourage the development of expressive and distinct semantic space, we regularize the latent vectors with an orthogonal regularization. the empirical prior is derived by summarizing the distribution of the latent vectors from the training data while respecting the modeling capability of the language model itself, and we employ an online method to estimate the empirical prior for efficiency. we conduct experiments on the standard benchmarks used to evaluate text vae in terms of reconstruction perplexity and evaluation of generated samples. whether using deterministic latent vectors or sampling the latent vectors from the empirical posterior, the perplexity of the ae is lower than that of the vae models. the underlying reason is that not adding a simplistic prior avoids interfering with the language model's own capability, greatly increasing fidelity. in terms of generation, the text generated using sampling from the empirical prior distribution outperforms the state-of-the-art vae models (e.g., della @xcite ) on quality and diversity. the analyses demonstrate that the ae model requires fewer latent variables, and the visualization shows that the ae model learns a similar latent space structure to della. we will release the code upon acceptance. in nlp, text vae leverages various intriguing characteristics of the latent space @xcite @xcite . however, the modeling capacity and empirical performance are limited primarily due to kl vanishing. in such cases, the decoder disregards the latent space entirely and degenerates into a simplified language model. various training techniques have been proposed such as annealing @xcite @xcite , kl thresholding @xcite , and the combination with pretrained transformer models @xcite . @xcite pretrain the inference network with an autoencoder objective. in addition, there have been efforts to propose more powerful prior distributions @xcite @xcite @xcite . for example, dprior @xcite uses learnable vectors (dictionary atoms) to introduce a data-driven prior, and dior-cvae @xcite employs a diffusion model to enhance the prior. nevertheless, there is a lack of research addressing the necessity of the prior distribution in this context. considering the inherent robustness of language models in effectively modeling and generating coherent text, allowing the prior distribution to adapt to the empirical language distribution may be more advantageous than enforcing the condition variables to conform to a fixed prior distribution. we are the first to demonstrate that removing the hardcoded prior makes the ae-based language models achieve better sample quality and diversity performance than the state-of-the-art vae transformers. @xcite first pretrain a big language model in the vae paradigm, and indicate its superior controllability and capability than casual language models. they strive to assist the research community in acknowledging the significance of latent conditional language modeling in pre-training and making it more feasible in practice. however, the training difficulty may limit the development of scaling up text vae. in comparison, ae is naturally suitable for large-scale pretraining, which can be traced back to the restricted boltzmann machines @xcite . the additional high-level guidance before the next token prediction can help to capture the underlying causal structure of the generative process. moreover, the learned latent space allows better controllability and interpretability. these advantages can be more appealing when further scaling up. ## human evaluation we conduct a human evaluation to compare samples generated by the ae model and the sota baseline della using an a/b test @xcite . we sample 200 sentences from the generated sentences of each model. we present three annotators with two samples, one from each model, and ask them to indicate their preference based on grammaticality, novelty, and overall quality ( ## case study visualization we employ t-sne to inspect the representation space of della, our ae, and gpt-2. using the yelp development set, we categorize 1-star sentences as negative and 5-star sentences as positive. figure 5 depicts clearer separation in della and ae's latent representations, indicating that the similar latent space to disentangle semantics. by contrast, the sentence representations of gpt-2 obtained by average token representations exhibit entangled semantic discernment (figure 6 in appendix), demonstrating the superiority of the conditional language models. interpolation samples of mid-point interpolation between two random sentence pairs are shown in figure 7 (appendix f). the interpolated sentences exhibit meaningful semantic interpolation. ## conclusion we investigated training latent conditional language models with autoencoding, which removes the prior and the corresponding restriction in vae. we found that adapting the prior distribution to the language model can satisfy the need for sampling without affecting the modeling capability of the language model itself. the generated samples from the empirical prior achieve better quality and diver-sity than the vae-based models. future work includes the extension of ae-based language models to large-scale pertaining and instruction fine-tuning, in which the high-level semantic condition of the target sequence may have additional benefits.