Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space (2020)

Abstract: When trained effectively, the Variational Autoencoder (VAE) can be both a powerful generative model and an effective representation learning framework for natural language. In this paper, we propose the first large-scale language VAE model Optimus (Organizing sentences via Pre-Trained Modeling of a Universal Space). A universal latent embedding space for sentences is first pre-trained on large text corpus, and then fine-tuned for various language generation and understanding tasks. Compared with GPT-2, Optimus enables guided language generation from an abstract level using the latent vectors. Compared with BERT, Optimus can generalize better on low-resource language understanding tasks due to the smooth latent space structure. Extensive experimental results on a wide range of language tasks demonstrate the effectiveness of Optimus. It achieves new state-of-the-art on VAE language modeling benchmarks.

## introduction pre-trained language models (plms) have substantially advanced the state-of-the-art across a variety of natural language processing (nlp) tasks @xcite @xcite @xcite @xcite . plms are often trained to predict words based on their context on massive text data, and the learned models can be fine-tuned to adapt to various downstream tasks. plms can generally play two different roles: (i) a generic encoder such as bert @xcite to provide contextualized representations for language understanding tasks, and (ii) a powerful decoder such as gpt-2 @xcite to generate text sequences in an auto-regressive manner. in a bid to combine language understanding and generation tasks in one unified framework, several model variants have been proposed, including unilm @xcite , bart @xcite , and t5 @xcite . although significant performance improvement has been reported on a wide range of nlp tasks, these models lack of explicit modeling of structures in a compact latent space, rendering it difficult to control language generation/representation from an abstract level. variational autoencoders (vaes) @xcite provide a tractable method to train latent-variable generative models. in nlp, latent variables may assume the role of higher-level sentence representations, which govern a lower-level word-by-word generation process, thus facilitating controlled text generation @xcite . by representing sentences in a low-dimensional latent space, vaes allow easy manipulation of sentences using the corresponding compact vector representations, such as feature regularization specified by prior distributions, and guided sentence generation with interpretable vector operators. despite the attractive theoretical strengths, the current language vaes are often built with shallow network architectures, such as two-layer lstms @xcite . this limits the model's capacity and leads to sub-optimal performance. in this paper, we propose optimus, the first large-scale pre-trained deep latent variable models for natural language. optimus is pre-trained using the sentence-level (variational) auto-encoder objectives on large text corpus. this leads to a universal latent space to organize sentences (hence named optimus). optimus enjoys several favorable properties: (i) it combines the strengths of vae, bert and gpt, and supports both natural language understanding and generation tasks. (ii) comparing to bert, optimus learns a more structured semantic space due to the use of the prior distri-bution in training. as a result, the language representations learned by optimus are more universal / general in that they can be more easily adapted to a new domain/task. (iii) different from gpt-2, which generates human-like text but may lack effective means of controlling its high-level semantics (such as tense, topics, sentiment), optimus can be easily deployed for guided text generation. the effectiveness of optimus has been demonstrated with extensive experiments on language modeling, dialog response generation, text style transfer and low-resource language understanding. it achieves lower perplexity than gpt-2 on standard benchmarks, produces strong performance on guided text generation, and improves bert on feature-based language understanding tasks. the code and pretrained models are released on github foot_1 . along the way to build the first big vae language model, there are several technical contributions/implications that are novel: (i) latent vector injection: this work demonstrates two schemes to discuss how to effectively inject conditioning vectors into gpt-2 without re-training it. (ii) the design idea to combine bert/gpt-2 serves as a practical recipe to inspire people to integrate and reuse existing plms for larger and complex models. (iii) pre-training on massive datasets itself is an effective approach to reduce kl vanishing, as demonstrated by the state of-the-art performance on four vae language modeling datasets. (iv) the proof of vae objective from the lens of ib, showing that vae is a principled approach to balance the compactness and usability of learned representations. (v) improved performance on several language tasks shows the importance and necessity of pre-training a latent space. latent variable language modeling. language vaes have inspired new applications in nlp, via exploiting many interesting properties of the model's latent space @xcite . its modeling capacity and empirical performance is somewhat limited, partially due to the kl vanishing issue described in section 4.3. several attempts have been made to alleviate this issue, including different kl annealing/thresholding schemes @xcite @xcite , decoder architectures @xcite , auxiliary loss @xcite , semi-amortized inference @xcite , aggressive encoder training schedule @xcite , batch normalized inference @xcite and flexible posterior @xcite . @xcite have shown some promise that general encoder can benefit language generation. transformers @xcite are recently considered in vaes for classification @xcite and storytelling @xcite . pre-training vaes has been recently considered in conditional text generation to amortize the training of decoders and to allow easy adaptation in new generation tasks @xcite . all these efforts utilize simple lstm @xcite and shallow transformer @xcite architectures, thus with limited capacity. our paper is the first big vae model at the same scale of recent plms such as bert and gpt-2. more importantly, we show that pre-training a meaningful latent space on a large text corpus can largely reduce the kl vanishing issue, and lead to new state-of-the-art performance. ## background on nlms & gpt-2 to generate a text sequence of length t , @xmath0, • • • , x t ], neural language models (nlm) @xcite generate every token x t conditioned on the previous word tokens: ## experimental results we consider to apply the pre-trained optimus models to three types of downstream tasks: (i) language modeling, where optimus is compared with sota vae methods and gpt-2. (ii) guided language generation, where optimus shows its unique advantage in producing controllable sentences in contrast to gpt-2. (iii) low-resource language understanding, where the learned structured latent features can be used for fast adaptation in new tasks. ## discussion we present optimus, a large-scale pre-trained deep latent variable model for natural language. it introduces a smooth and universal latent space, by combining the advantages of vaes, bert and gpt-2 in one model. experimental results on a wide range of tasks and datasets have demonstrated the strong performance of optimus, including new state-of-the-art for language vaes. there are several limitations in current opti-mus. first, our pre-trained language vae is still under-trained due to limited compute resource, as the training reconstruction loss can still decrease. one may further train the models with higher latent dimension and longer time to fully release the power of pre-trained latent spaces. second, the current model can only control sentences of moderate length. one future direction is to consider more sophisticated mechanisms to gain stronger controlability over longer sentences while maintaining the compactness of latent representations. while deep generative models (dgms) such as vaes are theoretically attractive due to its principle nature, it is now rarely used by practitioners in the modern pre-trained language modeling era where bert/gpt dominate with strong empirical performance. that's why this paper makes a timely contribution to making dgms practical for nlp. we hope that this paper will help renew interest in dgms for this purpose. hence, we deliberately keep a simple model, believing that the first pretrained big vae model itself and its implications are novel: it helps the community to recognize the importance of dgms in the pre-training era, and revisit dgms to make it more practical. indeed, optimus is uniquely positioned to learn a smooth latent space to organize sentences, which can enable guided language generation compared with gpt-2, and yield better generalization in lowresource language understanding tasks than bert.