Controllable Natural Language Generation with Contrastive Prefixes (2022)

Abstract: To guide the generation of large pretrained language models (LM), previous work has focused on directly fine-tuning the language model or utilizing an attribute discriminator. In this work, we propose a novel lightweight framework for controllable GPT2 generation, which utilizes a set of small attribute-specific vectors, called prefixes (Li and Liang, 2021), to steer natural language generation. Different from Li and Liang (2021), where each prefix is trained independently, we take the relationship among prefixes into consideration and train multiple prefixes simultaneously. We propose a novel supervised method and also an unsupervised method to train the prefixes for single-aspect control while the combination of these two methods can achieve multi-aspect control. Experimental results on both single-aspect and multi-aspect control show that our methods can guide generation towards the desired attributes while keeping high linguistic quality.

## introduction the goal of controllable natural language generation (nlg) is to guide generation towards the desired attributes in the concerned aspects of the text. for example, the aspect can be topic or sentiment, and sentiment may have two attributes: positive and negative. previous work has focused on directly fine-tuning the existing models @xcite @xcite or using a discriminator to guide generation @xcite @xcite . @xcite achieves controllability at the expense of training a large conditional lm. gedi @xcite ) also trains conditional lms but uses them as discriminators to guide generation, introducing additional 345m parameters. besides, gedi focuses on single-aspect control, ignoring the need for multi-aspect control. pplm @xcite guides generation by iteratively updating the lm's hidden activations. however, this decoding strategy is extremely computationally intensive, resulting in a slow generation speed @xcite . @xcite proposes to optimize a prefix, which is a small continuous taskspecific vector, as a lightweight alternative to finetuning an nlg task, such as table-to-text generation or summarization. inspired by @xcite , we propose to use prefixes, a set of small continuous attribute-specific vectors, to steer nlg. compared with using an attribute model or a generative discriminator @xcite , using learned prefixes to achieve controllability has the following benefits. first, it introduces fewer additional parameters (~0.2%-2% of gpt2 parameters in our experiments). second, using prefixes keeps the inference speed comparable to that of the original gpt2 model. in a general sense, prefix-tuning (li and liang, 2021) can be considered as controlling the genera-tion of language models. prefix-tuning views each prefix as an independent control task thus trains each prefix separately (top in figure 1 ). however, one aspect of controllability in nlg involves multiple attributes, which might have a relationship with each other. for example, the sentiment aspect usually has two attributes: positive and negative, which are in opposition to each other. we think that this opposite relationship can be helpful to improve the controllability of a prefix. therefore, we propose a novel supervised method and a novel unsupervised one in our framework, which takes the relationship among prefixes into consideration and trains multiple prefixes simultaneously with novel training objectives, as illustrated in figure 1 . experimental results on the single-aspect control tasks (sentiment control, detoxification, and topic control) show that our proposed methods can guide generation towards the target attribute while keeping high linguistic quality, even when only several dozen labeled examples are available. in addition to single-aspect control, multi-aspect control can be achieved by combining the proposed supervised method with the unsupervised method in our framework. experimental results on the sentiment and topic control show that the prefixes trained with our method can successfully control these two aspects simultaneously. our main contributions are as follows: ficler and goldberg (2017) control the stylistic aspects of the generated text with a conditioned rnn (recurrent neural network) lm. holtzman et al. (2018) compose a committee of discriminators to guide an rnn generator towards the generations with the desired linguistic quality. hu et al. (2017) aim at controlling the sentiment and tense of the generated text by combining variational autoencoders (vae) and attribute discriminators. more recently, with the advent of transformers and large pretrained language models, such as gpt2, an extensive body of work has focused on controlling the generation of these transformerbased models. keskar et al. ( 2019 ) train a 1.63 billion-parameter conditional transformer lm from scratch with 55 attribute control codes to guide generation. however, this method is expensive and lacks flexibility since the control codes are fixed. @xcite address these limitations by developing a plug-and-play model which leverages an attribute discriminator to perturb the lm's hidden activations. however, updating gradients at the token level results in slow inference. instead of updating the hidden activations, @xcite ; @xcite ; @xcite introduce generative discriminators to re-weight the next token distributions on the fly during inference, thus improving the inference speed. our work is mostly related to @xcite ; li and liang (2021). @xcite use a pretrained lm followed by an attribute alignment function to encode the tokens of the target attributes and the resulting hidden states are used to control generation. different from their work, we do not take the tokens of the target attributes as input. instead, we directly train a set of parameters, which acts as the prepended hidden states of gpt2, to control generation. avoiding using attribute tokens can circumvent the problems when it is difficult to describe the desired attribute with only one word. besides, @xcite focus on attributes disentanglement, which is not a focus in our work, so our training methods are different. prefix-tuning (li and liang, 2021) can, in a general sense, be viewed as controlling the generation of lms, where the lm is controlled to depict a specific nlg task, while in this work, the lm is controlled to carry specific attributes in a generation. besides, our proposed methods for prefix training are different from li and liang (2021), as stated in section 1. ## method our method uses prefixes to guide gpt2 generation, where a prefix is a continuous attributespecific vector prepended to the activations of gpt2. prefixes are free parameters denoted as h θ . different from li and liang (2021), where each prefix is trained independently, we consider the relationship among attributes and train multiple prefixes simultaneously, so h θ is of dimension n × m × d, where n is the number of prefixes. in single-aspect control, n equals the number of attributes in the concerned aspect. m is the length of a prefix. @xmath0, where l is the number of transformer layers, e is the hidden size, and 2 indicates one key vector and one value vector. following li and liang (2021), we reparametrize h θ [i, j, :] = w i h θ [i, j, :] by a smaller parameter (h θ ) composed with a large matrix (w i ). after the training finishes, only h θ needs to be saved for generation while w and h θ can be discarded. since the gpt2 parameters are kept frozen during training, they do not need to be saved either. figure 2 shows an example of the generation process under the control of a trained prefix. the prefixes can be trained in a supervised, semi-supervised, or unsupervised way. since the semi-supervised method is a combination of the supervised and the unsupervised method, we introduce the supervised and the unsupervised method in this section. for clarity, we introduce these methods under the single-aspect control setting. ## experiments we experiment with three tasks: sentiment control, detoxification, and topic control. we compare our method to gpt2, pplm, and gedi. we focus on english text in all the experiments and we experiment with gpt2-medium (345m parameters) for all the methods. we use the original implementation of pplm and gedi released by @xcite and @xcite , and the hyperparameters are set to the reported value in the original paper. the detailed hyperparameters in each task are listed in appendix a. for the gpt2 model, we do experiments under two settings. first, the gpt2 model generates completions of each prompt in the evaluation dataset, which is denoted as gpt2medium. second, gpt2-medium + prompt engineering prepends a guiding sentence to each testing prompt and then generates completions of each augmented prompt. we evaluate the linguistic quality and attribute alignment of the generation. the linguistic quality is evaluated using the perplexity calculated by gpt2-large (774m parameters). to evaluate the robustness of our supervised method with the size of the training dataset, we experiment with the following three different settings: 1) using the complete training dataset; 2) using 1,000 examples per attribute for training; 3) using 24 examples per attribute for training. we evaluate our unsupervised method on the sentiment control task and the detoxification task, which are binary tasks. note that different from the supervised method, our unsupervised method does not use any attribute labels, so the order of the attributes in the trained prefixes is undetermined. after the prefixes finish training using the unsupervised method, we manually check the order of the attributes. ## conclusion we propose a novel framework for controllable gpt2 generation with frozen lms, which utilizes contrastive prefixes to guide generation. experimental results show that our framework can not only successfully guide generation from a single aspect but also achieve promising results on multiaspect control tasks. besides the control tasks we experimented with, our proposed framework can be freely applied to other desired attributes.