Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoders (2020)

Abstract: Conditional Text Generation has drawn much attention as a topic of Natural Language Generation (NLG) which provides the possibility for humans to control the properties of generated contents. Current conditional generation models cannot handle emerging conditions due to their joint end-to-end learning fashion. When a new condition added, these techniques require full retraining. In this paper, we present a new framework named Pre-train and Plug-in Variational Auto-Encoder (PPVAE) towards flexible conditional text generation. PPVAE decouples the text generation module from the condition representation module to allow “one-to-many” conditional generation. When a fresh condition emerges, only a lightweight network needs to be trained and works as a plug-in for PPVAE, which is efficient and desirable for real-world applications. Extensive experiments demonstrate the superiority of PPVAE against the existing alternatives with better conditionality and diversity but less training effort.

## introduction currently, neural generation techniques have powered many inspiring applications, e.g., poem generation @xcite , neural machine translation (nmt) @xcite and chatbot @xcite . conditional (also known as controllable) text generation is an important task of text generation, aiming to generate realistic text that carries a specific attribute (e.g., positive or negative sentiment). a common solution is to encode the condition into a vector representation and then integrate it with the text generation process @xcite @xcite . these existing neural models have achieved encouraging results. however, when a new condition is added (e.g., a new topic for categorical generation), they require a full retraining or finetuning. this process is both time-consuming and computationally inefficient @xcite . both fine-tuning and retraining are not desirable in real-world applications since the delivery (e.g., transmitting updated weights through the internet) and client-side re-deployment (e.g., distribute updated weights to users) of large-scale weights are often difficult. inspired by the recent success of variational auto-encoder (vae) @xcite based post-hoc conditional image generation strategy @xcite , we provide a new perspective for flexible conditional text generation. we propose pre-train and plug-in variational auto-encoder (ppvae), which decouples the text generation module from the condition representation module. ppvae is a hierarchical framework composed of two vaes: (1) pretrainvae, which derives the global latent space of text with its encoder (pre-trained global encoder) and learns to generate text based on an easily-accessible large unlabeled dataset with its decoder (pre-trained global decoder); (2) pluginvae, which is a lightweight neural network that learns to transform vectors from the conditional latent space to the global latent space, and vice versa. this mapping function can be easily learned with only a few conditional training samples. in this sense, once we transform a latent variable (also known as latent code) randomly sampled from the conditional space distribution to the global space, the pre-trained global decoder is directly adopted for generation. in other words, whenever a new condition emerges, we only need to train a pluginvae and directly plug it into the framework. different from the existing end-to-end neural models @xcite @xcite , ppvae focuses on the learning of pure transformation between the continuous latent spaces, instead of the tricky discrete text generation. once trained, pretrainvae is fixed for text representation and generation under all conditions. our proposed framework decouples the conditional space learning from the text generation, endowing ppvae with more flexibility when handling emerging conditions. also, training only a small conditional network for latent space transformation is much more efficient than co-training with the text generation. additionally, we can easily increase the capability of generation using a larger corpus or deeper neural networks for text encoding and decoding. our main contributions can be summarized as follows: (1) we propose a novel framework, ppvae, for conditional text generation, which allows a separate training for a new condition without retraining the whole network. (2) we conduct extensive experiments and analysis to verify the effectiveness of our proposed ppvae. our framework achieves state-of-the-art performance on conditionality in both automatic and human evaluations. ## related work boosted by the recent success of deep learning technology, natural language generation (nlg) has recently become popular in the nlp community. many great works have attempted to solve various subtasks like dialogue generation @xcite , poetry generation @xcite and story generation @xcite and new techniques keep emerging @xcite @xcite . however, due to the blackbox nature of neural networks, the recent proposed generic models suffer the problem of lacking interpretability and controllability. to handle this problem and support generating plausible text with a specified condition, conditional text generation @xcite @xcite has recently attracted extensive attention. current research in this direction mainly falls into two fashions: the supervised methods and semi-supervised methods. for supervised methods, @xcite ; @xcite first converted the condition information to one-hot vectors, then integrated them into a generator and a discriminator. to enhance the correlation between structured conditional code and generated samples, @xcite adopted an extra adversarial classifier to infer the structured code from generated samples. @xcite used multiple generators for multiple conditions and a multi-class classifier to provide training signals for the learning of generators. however, given only a limited number of conditional samples, semi-supervised methods are compulsory. to utilize the implicit conditional distribution behind the unlabeled text, @xcite introduced a classifier into the vae architecture. @xcite further involved two additional independent regularization terms in enhancing the disentanglement between structured code and unstructured code. very recently, @xcite used human-defined "control code" to pre-trained language model in an unsupervised manner. our work falls in the category of semisupervised learning yet differs from the existing works in the following ways: (1) our model decouples the text generation module from the condition representation module which two are tightly fused as a single one in previous studies, enabling possible exploitation for pre-trained language models (e.g., gpt-2 @xcite ). (2) our model allows single-condition generation, which could inspire new applications like polite speech generator @xcite and data augmentation @xcite . (3) our model can handle emerging conditions while achieving state-of-theart performance with fewer parameters and less training time. ## preliminaries variational auto-encoder (vae). vae (kingma and ba, 2015) is widely used in continuous generation (e.g., image generation). @xcite introduced vae to nlg to solve the "one-to-many" generation problem (i.e., generating multiple feasible samples for the same input). given a latent variable z randomly sampled from a prior distribution, vae comprises an encoder enc(x) = q φ (z|x) and a decoder dec(z) = p θ (x|z). the encoder aims to encode input data x into latent space @xmath0. the decoder is used to reconstruct the original input x, given the corresponding z. thus, the loss function of vae is formulated as: the first term ensures that vae can distill compact variable z in latent space for reconstruction. the second term pushes posterior distribution to be close to the prior distribution, securing the mutual information between original data and the latent space @xcite . ## pre-train and plug-in variational auto-encoder as a basis for semi-supervised learning, a large unlabeled corpus should include diverse text which covers a vast spectrum of conditions. thus, text under each condition forms a conditional latent space, which could be mapped from a larger global latent space. based on this, we propose a pretrain-vae and a pluginvae to derive the global and conditional latent space, respectively.