Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation (2022)

Abstract: In this paper, we propose a variational autoencoder with disentanglement priors, VAE-Dprior, for task-specific natural language generation with none or a handful of task-specific labeled examples. In order to tackle compositional generalization across tasks, our model performs disentangled representation learning by introducing a conditional prior for the latent content space and another conditional prior for the latent label space. Both types of priors satisfy a novel property called \epsilon -disentangled. We show both empirically and theoretically that the novel priors can disentangle representations even without specific regularizations as in the prior work. The content prior enables directly sampling diverse content representations from the content space learned from the seen tasks, and fuse them with the representations of novel tasks for generating semantically diverse texts in the low-resource settings. Our extensive experiments demonstrate the superior performance of our model over competitive baselines in terms of i) data augmentation in continuous zero/few-shot learning, and ii) text style transfer in the few-shot setting.

## introduction task-specific natural language generation (nlg) aims to generate texts that satisfy desired attributes of target tasks, such as text style transfer @xcite and task-specific data augmentation @xcite . herein, a task includes a set of taskspecific labels, optionally a set of labeled texts for that task @xcite . although there is already a large amount of labeled data for various tasks, in many application scenarios, such as ai assistants for legal aid, the labeled data of new tasks are still difficult to acquire. as a result, there may be no or just a handful of labeled texts for target tasks. in such a low-resource setting, given a new task, it is desirable to i) identify which information in texts is task-specific and which is task-independent, and ii) systematically and consistently combine the label representations of the new task with task-independent content representations for text generation. as illustrated in fig. 1 , data augmentation needs to combine content representations from seen tasks with novel task labels. in contrast, text style transfer requires combining the content representations extracted from inputs with target styles. most prior work assumes access to labeled data for supervised training. however, those models trained on seen tasks cannot generalize well to new tasks during inference @xcite . one of the key reasons is that the parameters of supervised models are tied to seen tasks such that a significant amount of fine-tuning data is needed for adapting to new tasks. for prompt-based and guided decoding methods @xcite , although they require significantly less training data, it is still challenging to generate a large number of semantically diverse and coherent texts for new tasks in a robust way because they cannot well leverage the rich contents of the seen tasks. the key challenge of low-resource task-specific nlg is to disentangle content representations from label representations with few labeled data of target tasks. if content representations still contain taskspecific information from seen tasks, they may well mislead the language generator after fusing with the representations of new tasks. prior works tackle this problem by enforcing the random variables of content representations to be independent of those of label representations @xcite . however, in practice, both types of random variables are not always independent. for example, the random variables of emotion labels naturally depend on the contents of the events causing them. in this work, we propose a deep vae model with novel disentanglement priors, coined vae-dprior, for task-specific natural language generation in the zero-shot and few-shot settings. in contrast to the widely used unconditional priors in the vae framework, the new priors are conditional, satisfying a novel property called ϵ-disentangled, which motivates a new way of regularization for disentangling representations without forcing independence between the corresponding random variables. the new priors build a constraint space for latent content representations and latent label representations with the aims to i) minimize information overlap between the two types of representations and ii) enable generalization across tasks with little labeled training data. one of the priors is a conditional gaussian mixture in the content subspace for sampling rich content representations without accessing original training data. another type of priors is a conditional multivariate gaussian per label that associates latent label representations with task-specific information, requiring only a label name or a small set of labeled examples. extending a pre-trained language decoder based on the prefix-tuning technique @xcite with those priors, our model is able to sample rich content representations of seen labels and combine them with the representations of new labels to generate diverse and natural sentences. in addition, we empirically observe that vae-dprior alleviates posterior collapse @xcite , which is a long-standing problem of vaes that makes it difficult to train a latent model to generate coherent and semantically diverse texts. to sum up, our key contributions are threefold: i) we propose a vae-dprior model with novel disentanglement priors for low-resource taskspecific nlg tasks. it enables sampling diverse content representations directly from the content prior; ii) we introduce ϵ-disentangled, which sets a novel regularization goal for disentangled representations; iii) our model outperforms competitive baselines in the low-resource settings on the tasks of text style transfer and data augmentation for continual few/zero shot text classification. ## methodology to tackle task-specific nlg tasks in low-resource settings, we introduce a deep generative model vae-dprior, which employs disentanglement priors, including a content prior for rich contents, to generate coherent and semantically diverse texts. we are provided with a large corpus of labeled sentences d (0) = {x i , y i } n @xmath0, where a sentence x @xmath1. the goal is to learn a single model that can generate diverse texts for any new task or a sequence of k distinct new tasks {t (1) , t (2) ,...,t (k) }. each new task includes multiple novel labels, where a label @xmath2, y i } m @xmath3. the model is evaluated on both data augmentation for continual text classification described in sec. 3 and few-shot text style transfer detailed in appendix c.1. for evaluating data augmentation, a text classifier is trained sequentially on k new tasks and evaluated on the test sets of all seen tasks {t (1) ,...,t (t) } till time t. for each task, its training data includes the texts generated by the nlg models in order to evaluate to what degree the augmented texts improve the classifier performance. in the zero-shot setting, the classifier is trained only on the generated texts using the label names of new tasks, while the support sets are also used for data augmentation and classifier training in the few-shot setting. ## experiments we evaluate our model on both continual few/zero shot text classification and few-shot text style transfer. the former requires sampling rich content representations from seen tasks, while the latter expects to retain task-independent contents from inputs. in both cases, it is desirable for models to systematically combine latent label and content representations across tasks in a consistent manner. the details of few-shot text style transfer are available in appendix c. we compare vae-dprior with five style-transfer baselines and show superior results on two datasets in the few-shot setting in terms of accuracy of style transfer, semantic relevance and naturalness of the generated text. ## conclusion in this work, we propose a vae model with disentanglement priors for disentangled representation learning in low resource controllable nlg tasks. the disentanglement priors satisfy a novel property called ϵ-disentangled which builds a constraint space for the training problem. this model is able to effectively combine rich content representations sampled from a conditional content prior and taskspecific representations for new tasks. its empirical performance outperforms the baselines on continual zero-shot/few-shot text classification and fewshot text style transfer by a wide margin.