Control, Generate, Augment: A Scalable Framework for Multi-Attribute Text Generation (2020)

Abstract: We introduce CGA, a conditional VAE architecture, to control, generate, and augment text. CGA is able to generate natural English sentences controlling multiple semantic and syntactic attributes by combining adversarial learning with a context-aware loss and a cyclical word dropout routine. We demonstrate the value of the individual model components in an ablation study. The scalability of our approach is ensured through a single discriminator, independently of the number of attributes. We show high quality, diversity and attribute control in the generated sentences through a series of automatic and human assessments. As the main application of our work, we test the potential of this new NLG model in a data augmentation scenario. In a downstream NLP task, the sentences generated by our CGA model show significant improvements over a strong baseline, and a classification performance often comparable to adding same amount of additional real data.

## introduction recently, natural language generation (nlg) has become a prominent research topic in nlp due to its diverse applications, ranging from machine translation (e.g., @xcite ) to dialogue systems (e.g., @xcite ). the common goal of these applications using automatic text generation is the augmentation of datasets used for supervised nlp tasks. to this end, one of the key demands of nlg is controlled text generation, more specifically, the ability to systematically control semantic and syntactic aspects of generated text. most previous approaches simplify this problem by approximating nlg with the control of one single attribute of the text, such as sentiment or formality (e.g., @xcite , @xcite , and @xcite ). however, the problem of controlled generation typically relies on multiple components such as lexical, syntactic, semantic and stylistic aspects. therefore, the simultaneous control of multiple attributes becomes vital to generate natural sentences suitable for downstream tasks. methods such as the ones presented by @xcite and @xcite succeed in simultaneously controlling various attributes. however, these methods depend on the transformation of input reference sentences, or do not scale easily to more than two attributes due to architectural complexities, such as the requirement for separate discriminators for each additional attribute. in light of these challenges, we propose the control, generate, augment framework (cga), a powerful model to synthesize additional labeled data sampled from a latent space. the accurate multi-attribute control of our approach offers significant performance gains on downstream nlp tasks. we provide the code and all generated english sentences to facilitate future research 1 . the main contribution of this paper is a scalable model which learns to control multiple semantic and syntactic attributes of a sentence. the cga model requires only a single discriminator for simultaneously controlling multiple attributes. to the best of our knowledge, we are the first to incorporate techniques such as cyclical word-dropout and a context-aware loss, which allow the cga model to generate natural sentences given a latent representation and an attribute vector, without requiring an input reference sentence during training. we present automatic and human assessments to confirm the multi-attribute control and high quality of the generated sentences. further, we provide a thorough comparison to previous work. we use cga as a natural language generation method for data augmentation, which boosts the performance of downstream tasks. we present data augmentation experiments on various english datasets, where we significantly outperform a strong baseline and achieve a performance often comparable to adding same amount of additional real data. ## method we now present our model for controlled text generation. our model is based on the sentence-vae framework by @xcite . however, we modify this model to allow the generation of sentences conditioned not only on the latent code but also on an attribute vector. we achieve this by disentangling the latent code from the attribute vector, in a similar way as the fader networks @xcite , originally developed for computer vision tasks. as we will see, this simple adaption is not sufficient, and we introduce further techniques to improve the multi-attribute sentence generation. ## comparison to previous work as a final analysis, we compare our results with previous state-of-the-art models for both singleattribute and multi-attribute control. ## conclusion to the best of our knowledge, we propose the first framework for controlled natural language generation which (1) generates coherent sentences sampling from a smooth latent space, with multiple semantic and syntactic attributes; (2) works within a lean and scalable architecture, and (3) improves downstream tasks by synthesizing additional labeled data. to sum up, our cga model, which combines a context-aware loss function with a cyclical worddropout routine, achieves state-of-the-art results with improved accuracy on sentiment, verb tense and person number attributes in automatic and human evaluations. moreover, our experiments show that our cga model can be used effectively as a data augmentation framework to boost the performance of downstream classifiers. a thorough investigation of the quality of the attribute-invariant representation in terms of inde-pendence between the context and the attribute vector will provide further insights. additionally, a benchmark study of the maximum possible length of the generated sentences and the number of controllable attributes will deepen our understanding of the capabilities and limitations of cga. dimensions (for three attributes) @xcite @xcite . for training the decoder we set the initial hidden state as @xmath0. moreover, we use teacher-forcing combined with the cyclical worddropout described in equation 4 .