Controllable Text Generation with Focused Variation (2020)

Abstract: This work introduces Focused-Variation Network (FVN), a novel model to control language generation. The main problems in previous controlled language generation models range from the difficulty of generating text according to the given attributes, to the lack of diversity of the generated texts. FVN addresses these issues by learning disjoint discrete latent spaces for each attribute inside codebooks, which allows for both controllability and diversity, while at the same time generating fluent text. We evaluate FVN on two text generation datasets with annotated content and style, and show state-of-the-art performance as assessed by automatic and human evaluations.

## introduction recent developments in language modeling @xcite @xcite @xcite ) make it possible to generate fluent and mostly coherent text. despite the quality of the samples, regular language models cannot be conditioned to generate language depending on attributes. conditional language models have been developed to solve this problem, with methods that either train models given predetermined attributes @xcite , use conditional generative models @xcite , fine-tune models using reinforcement learning @xcite , or modify the text on the fly during generation @xcite . as many researchers noted, injecting style into natural language generation can increase the naturalness and human-likeness of text by including pragmatic markers, characteristic of oral language @xcite @xcite ). text generation with * work done while at uber ai labs. conditional vaes (cvae) @xcite ) and their variants have been adopted for the task and are able to generate diverse texts, but they suffer from posterior collapse and do not strictly follow the given attributes because their latent space is pushed towards being a gaussian distribution irrespective of the different disjoint attributes, conflating the given content and style. an ideal model would learn a separate latent space that focuses on each attribute independently. for this purpose, we introduce a novel natural language generator called focused-variation network (fvn) foot_0 . fvn extends the vector-quantised vae (vq-vae) @xcite , which is non-conditional, to allow conditioning on attributes (content and style). specifically, fvn: text 1 name slot is a eattype slot with a customerrating slot rating. it is a pricerange slot priced food slot restaurant in area slot near near slot. it is familyfriendly slot. agreeable let's see what we can find on name slot. i see, well, it is an eattype slot with a customerrating slot rating, also it is a pricerange slot priced food slot restaurant in area slot and near near slot, also it is familyfriendly slot, you see? disagreeable i mean, everybody knows that name slot is an eattype slot with a customerrating slot rating. it is a pricerange slot priced food slot restaurant in area slot near near slot. it is familyfriendly slot. delex. text 2 name slot is a food slot place near near slot in area slot and pricerange slot priced. it has a customerrating slot rating. it is an eattype slot and familyfriendly slot kid friendly. conscientious let's see what we can find in name slot. emm ... it is a food slot place near near slot in area slot and pricerange slot priced. it has a customerrating slot rating. it is an eattype slot and familyfriendly slot. unconscientious oh god yeah, i don't know. name slot is a food slot place near near slot in area slot and pricerange slot priced. it has a customerrating slot rating. it is an eattype slot and familyfriendly slot kid friendly. we tested fvn on two datasets, personagenlg @xcite and e2e @xcite that consist of content-utterance pairs with personality labels in the first case, and the experimental results show that it outperforms previous state-ofthe-art methods. a human evaluation further confirms that the naturalness and conveyance of fvn generated text is comparable to ground truth data. ## related work our work is related to cvae based text generation @xcite @xcite , where the goal is to control a given attribute of the output text (for example, style) by providing it as additional input to a regular vae. for instance, the controlled text generation method proposed by @xcite extends vae and focuses on controlling attributes of the generated text like sentiment and style. differently from ours, this method does not focus on generating text from content meaning representation (cmr) or on diversity of the generated text. @xcite use a memory augmented cvae to control for persona, but with no control over the content. the works of @xcite @xcite on style-variation generators adopt sequence-to-sequence based models and use human-engineered features @xcite (e.g. personality parameters or syntax features) as extra inputs alongside the content and style to control the generation and enhance text variation. however, using human-engineered features is labor-intensive and, as it is not possible to consider all possible feature combinations, performance can be sub-optimal. in our work we instead rely on codebooks to memorize textual variations. there is a variety of works that address the problem of incorporating knowledge or structured data into the generated text (for example, entities retrieved from a knowledge base) @xcite , or that try generate text that is in line with some given story @xcite . none of these works focuses specifically on generating text that conveys content while at the same time controlling style. last, there are works such as @xcite that focus on generating text consistent with an emotion (aiming to create an empathetic agent) without, however, directly controlling the content. ## methodology our proposed fvn architecture (figure 1 ) has the goal to generate diverse texts that respect every attribute provided as controlling factor. we describe a specific instantiation where the attributes figure 1 : focused-variation network (fvn) has four encoders (text-to-content encoder, text-to-style encoder, content encoder and style encoder), two codebooks (for content and style), and one text decoder. the training data contains ground-truth text with associated content and style. the text decoder uses v c and v s , latent vectors of content and style, as well as the latent vectors e c k and e s n from codebooks (the nearest to the z c and z s vectors produced by the text-to-content and text-to-style encoders) to generate text back. to further control the content and style of the generated text, we feed the o l output vectors of the generated text t to text encoders (content and style). o l are aligned to a word embedding codebook. during training, fvn receives as input an intended content c and style s as well as a reference text t. the reference text is passed through two encoders (text-to-content and text-to-style), while content and style are encoded with a content encoder and a style encoder. the text-to-content encoder maps input text t into a content latent vector z c and the text-to-style encoder maps the input text t into a latent style vector z s . the closest vectors to z c and z s from the content codebook e c and style codebook e s , e c k and e s n , are selected. the content encoder encodes the intended content frame into a latent vector v c and the style encoder encodes the intended style into a latent vector v s . a text decoder then receives e c k , e s n , v c and v s and generates the output text t . the generated text is subsequently fed to a content and a style decoder that predict the intended content and style. at prediction time (figure 2 ), only content c and style s are given, and in order to obtain e c k and e s n without an input text, we a) collect a distribution over the codebook indices by counting, for each training datapoint containing a specific value for c and s, the amount of times a specific index is used, and b) sample e c k and e s n from these frequency distributions. these disjoint distributions allow the model to focus on specific content and style by using them for conditioning and the sampling allows for variation, hence the name of focused variation. v c and v s obtained from the content and style encoders and the sampled e c k and e s n are provided to the text generator that generates t . the rest of this section will detail each component and the training and prediction processes. ## experiments to test the capability of fvn to generate diverse texts that convey the content while adopting a certain style, we use the personagenlg text generation dataset for dialogue systems that contains cmr and style annotations. to test if fvn can convey the content (both slots and values) correctly on an open vocabulary, with complex syntactic structures and diverse discourse phenomena, we use the end-2-end challenge dataset (e2e), a text generation dataset for dialogue systems that is annotated with cmr. ## conclusion in this paper, we studied the task of controlling language generation, with a specific emphasis on content conveyance and style variation. we introduced fvn, a novel model that overcomes the limitations of previous models, namely lack of conveyance and lack of diversity of the generated text, by adopting disjoint discrete latent spaces for each of the desired attributes. our experimental results show that fvn achieves state-of-the-art performance on per-sonagenlg and e2e datasets and generated texts are comparable to ground truth ones according to human evaluators.