D is C up: Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generation (2022)

Abstract: Prompt learning with immensely large Casual Language Models (CLMs) has been shown promising for attribute-controllable text generation (CTG). However, vanilla prompt tuning tends to imitate training corpus characteristics beyond the control attributes, resulting in a poor generalization ability. Moreover, it is less able to capture the relationship between different attributes, further limiting the control performance. In this paper, we propose a new CTG approach, namely DisCup, which incorporates the attribute knowledge of discriminator to optimize the control-prompts, steering a frozen CLM to produce attribute-specific texts. Specifically, the frozen CLM model, capable of producing multitudinous texts, is first used to generate the next-token candidates based on the context, so as to ensure the diversity of tokens to be predicted. Then, we leverage an attribute-discriminator to select desired/undesired tokens from those candidates, providing the inter-attribute knowledge. Finally, we bridge the above two traits by an unlikelihood objective for prompt-tuning. Extensive experimental results show that DisCup can achieve a new state-of-the-art control performance while maintaining an efficient and high-quality text generation, only relying on around 10 virtual tokens.

## introduction attribute-controllable text generation (ctg) aims to produce texts that satisfy desired attributes (e.g., sentiment, topic, etc.), facilitating safer and more practical text generation applications. for example, we need to control the emotion and politeness of generated text in a dialogue system for a more friendly interaction, while it is also crucial to avoid generating mindless and offensive content such as racial discrimination and toxic words. current transformer-based pre-trained language models (plms), especially casual language models (clms) like the gpt family @xcite , have enabled generation of texts of an unprecedented quality. however, due to the lack of interpretability of deep neural networks, it is often hard to guarantee the controllability of these models @xcite . this is a challenging and largely unsolved problem. the most natural ways of using transformerbased plms for ctg are to fine-tune or retrain the models @xcite @xcite , so as to control the plms to generate texts satisfying the control attributes. such methods have achieved certain performance breakthroughs in this area. nevertheless, the scale of plms is getting larger in recent years, making the plms resourceintensive to fine-tune or retrain. therefore, increasing attention has been paid to the decoding-time methods, where a plm is always fixed and a guided module is used to steer the text generation process. the core idea of existing decoding-time approaches is to train an external guided module that consists of a discriminator @xcite @xcite @xcite or generative discriminator @xcite , to adjust the probability of naturally producing a token by the plm at the decoding phase. this type of method exhibits a strong controllability while maintaining the generalization ability of the original plm. however, those methods decouple the guided module from the generative plm, resulting in either computational challenges (i.e., longer inference time or additional parameters) or a negative impact on text generation quality (i.e., arbitrary output text with a lower perplexity). more recently, ctg approaches based on prompt-tuning are proposed @xcite . control-prompts are usually trained separately with some different attributespecific corpus, using maximum likelihood estimation (mle) on the traditional next-token prediction task (see more details in appendix a.2). the trained control-prompts are then used as a prefix to steer the attribute-specific generation. this mechanism manifests a promising text quality and a lower computational cost; however, it still suffers from the following drawbacks: (problem 1) the learned control-prompts may absorb the features of training corpus aimlessly and are easily overfitted to other aspects in the training data beyond the control attributes, such as domain style, resulting in the generation of monotonous text. for example, if the sentiment control prompts are optimized in the movie review data, then the generated texts are generally relevant to movies even given prompt text from other domains. the detailed examples are given in appendix 6. (problem 2) attribute control prompts are often trained independently with single-attribute corpus, resulting in the inability to capture the relationship between multiple attributes. however, the previous works @xcite @xcite reveal that attribute reference is an important factor in ctg. for example, when generating sentences toward a positive sentiment, we hope the model can refer to the negative ones, so as to achieve a better control performance. to tackle the aforementioned problems, we propose discup, a discriminator cooperative unlikelihood prompt-tuning approach for ctg. the key idea is to move the attribute-discriminator, which is usually used to re-rank candidate tokens in the decoding-time approaches, to the training phase for augmenting control-prompt learning, allowing the learned prompts to inherit the advantages of decode-time approaches to some extent. specifically, the knowledge of the attribute-discriminator is distilled into some continuous control-prompts by a novel objective, using unlikelihood training @xcite to steer a frozen clm to produce attribute-specific texts. different from maximizing the probability of the next token under a given context, the proposed objective contains two parts: a likelihood objective that encourages the model to generate candidate tokens satisfying the desired attributes, and an unlikelihood objective that allows the model to alleviate generating candidate tokens inconsistent with the target attributes. in our approach, the candidate tokens are natu-rally self-generated by the frozen clm (i.e., the top-k highly probable tokens under the given context), which is trained on a large corpus and capable of producing multitudinous texts, rather than the ground-truth tokens appearing in the training samples. this will help the control-prompts relieve the problem of over-fitting the features in training data other than target attributes, while ensuring that the ctg model does not deviate far from the original clm. as such, the aforementioned problem 1 can be addressed. meanwhile, re-ranking the candidate tokens by the attribute-discriminator to chose the desired/undesired tokens, would also indirectly incorporate the relationship between different attributes (thus addressing problem 2), and the unlikelihood training would further enhance the control performance of the ctg model. experimental results on two attributecontrollable generation tasks, i.e., sentiment control and toxicity avoidance, prove that discup can achieve a new sota control performance. in addition, our method shows a strong comprehension ability, which can simultaneously guarantee a better text quality and a lower computational cost. our main contributions are as follows: (1) we provide a new alternative for controllable generation, by moving the attribute-discriminator in the scheme of decoding-time approaches into the training phase for prompt learning. this allows the model to take advantages of both prompttuning and decoding-time methods. (2) we propose a novel unlikelihood training strategy to enhance the control-prompts learning, which uses the knowledge in the attribute-discriminator to select the likely/unlikely target tokens from the selfgenerated tokens of the frozen clm, instead of carrying out the traditional next-token prediction based on a training corpus. (3) we conduct extensive experiments on the tasks of sentiment control and toxicity avoidance, and the results prove the effectiveness of discup and at same time highlight the promise of prompt-learning in ctg. ## related work a large number of plm-based ctg approaches have recently emerged. the most natural way is to retrain/refactor the plms to establish ctgspecific models. ctrl @xcite is a representative early work, which trains a conditioned language model on the training corpus containing a variety of control code. @xcite propose an alignment function module to transform the attribute representation, and steer the gpt2-based text generation. cocon (content-conditioner) @xcite injects a control block into the gpt2 model and provides the control code as a separate input, then retrains the whole module by self-supervised learning. @xcite propose pointer, which modifies the structure of transformer to generate text in a progressive manner for lexically constrained text generation. @xcite propose a mention flags (mf) module, which is injected into the decoder of the transformer, to achieve a higher level of constraint satisfaction. however, retraining/refactoring the plms faces the challenge of lacking labeled data and high computational costs. as the size of plms rapidly increases, the retraining approaches become computationally expensive. therefore, decode-time approaches, in which the parameters of plm are fixed, become widely applied to ctg. pplm @xcite uses a simple attribute classifier on the head of a plm to update the hidden layer by gradient feedback, to achieve the goal of generating desired attribute texts. instead of updating the hidden layer, fudge @xcite directly uses the discriminator to select candidate tokens produced by a frozen gpt2 model. in order to accelerate the generation process, gedi @xcite trains the class-conditional language model (cclm) as generative discriminators to guide the generation from a base gpt2. plug-and-blend @xcite extends gedi to controllable story generation by introducing a planner module. similarly, dexpert @xcite fine-tunes gpt2 as expert (anti-expert) to re-rank the predictions of the plm. decoding-time approaches usually can achieve a competitive control effect, yet fall short in text quality and inference speed, since the guided modules are always decoupled from the generator. recently, prompt-based methods for ctg have been proposed. @xcite propose the use of prefix tuning, which freezes the plm's parameters and back-propagates the error to optimize a continuous task-specific vector to realize a controllable text generation. based on prefix-tuning, @xcite takes into consideration the relationship among prefixes and trains multiple prefixes simultaneously. @xcite leverage the prompt-tuning @xcite to establish a multi-attribution control framework, namely tai-lor. our work shows some similarity to tailor, but differs in that we focus on improving the inherent problems in the vanilla prompt tuning, instead of prompt-based multi-attribute control. to the best of our knowledge, we are the first to move the discriminator to the training phase, allowing the plms to learn a re-ranked token distribution by incorporating the attribute discriminator information, and optimize control-prompts for attribute-specific text generation. as a result, the proposed method inherits the advantages of both decode-time and prompt-tuning approaches to generate higher-quality texts, with a faster inference speed and a better control performance. director @xcite also combines the discriminator in training phase for text generation. however it adds extra classifier architecture to the standard decoder-layer, which deviates from the paradigm of prompt-tuning. ## problem definition in this paper, we are concerned with the task of attribute-controllable text generation that aims to steer a casual language model (clm) (i.e., gpt2, see more details in appendix a.1) to generate texts satisfying the attribute constraint (e.g., sentiment, topic, toxicity, etc.). concretely, given the prompt text x 1:@xmath0, x 2 , . . . x t }, the goal of our task is to generate the continuations x t+1:@xmath1, x t+2 , . . . , x n }, and let the whole text x 1:n conform to the target control attribute denoted as c. it can be formally described as: ## conclusions we have proposed a novel alternative for attributecontrollable text generation, namely discriminator cooperative unlikelihood prompt-tuning (dis-cup). instead of the traditional next-token prediction based on a training corpus, we use a attributediscriminator to select unlikely/likely candidates from the tokens naturally self-generated by a frozen clm, which is trained on multitudinous textual corpora and capable of producing texts with a high degree of diversity. then, unlikelihood training is employed to optimize the control-prompts. experiments conducted on two typical ctg tasks, i.e., sentiment control and toxicity avoidance, prove that our approach not only significantly outperforms the vanilla prompt-tuning approaches, but also exhibits superiority over the existing training and decodingtime approaches.