Harnessing the Plug-and-Play Controller by Prompting (2023)

Abstract: Controllable text generation is a growing field within natural language generation (NLG) that focuses on producing text that meets specific constraints in real-world applications. Previous approaches, such as plug-and-play controllers (PPCs), aimed to steer the properties of generated text in a flexible manner. However, these methods often compromised the integrity of the language model’s decoding process, resulting in less smooth text generation.Alternatively, other techniques utilized multiple attribute prompts to align the generated text with desired attributes, but this approach required prompt design for each attribute and was dependent on the size of the language model. This paper introduces a novel method for flexible attribute control in text generation using pre-trained language models (PLMs). The proposed approach aims to enhance the fluency of generated text by guiding the generation process with PPCs. The key idea is to dynamically adjust the distribution of generated text by modifying prompts, effectively constraining the output space of the language model and influencing the desired attribute. To enable smooth cooperation between the PLM and the PPC, our work innovativel proposes a new model fine-tuning method: Reinforcement Learning with Dynamic Adjust Feedback (RLDAF).This fine-tuning process adapts a small subset of the language model’s parameters based on the generating actions taken during the PPC control process. The resulting harmonious collaboration between the PLM and PPC leads to improved smoothness in text generation during inference. Extensive experiments were conducted on the SST2 dataset, and the proposed method outperformed previous approaches in various evaluation metrics, including text fluency and attribute consistency.

## introduction enough studies have shown that large-scale plms can largely improve the performance of downstream tasks @xcite . these models can generate fluent text which is close to the human level @xcite through simple pretraining tasks on a large number of unlabeled text. plms are also capable of making the generated text meet the specific constraints in real applications, which has become a hot research field in natural language processing @xcite . to prevent fine-tuning the massive parameters of plms, plug-and-play controllers (ppc) are proposed to dynamically control the specific attributes of the generated text by an external module. for example, pplm @xcite uses an external attribute discriminator to guide and modify a small portion of parameters in plms. however, we found that since the core parameters of plms have to be changed every time a token is generated, this method destroys the integrity of plms, making the results easy to fall into the local optimum, and thus generating repetitive and meaningless text. another method used to control text generation is prefix-tuning @xcite . this method controls text generation by inserting a trainable prefix parameter before the model input, but the prefix parameter of this method is fixed once it is trained, which is difficult to be used for complex constraint control tasks. therefore, in this work, we propose a novel method to improve the smoothness of ppc-guided text generation. this method inserts a set of trainable prompt parameters at the beginning of the input sequence of plm and tunes a small portion of lm parameters to make the lm adapted to the external controller. in detail, during the fine-tuning phase, we generate the text under the control of the external attribute discriminator, which adjusts the parameters of prompts in each timestep. then, we calculate a reward according to the generated results to evaluate the generation quality, enabling the language model to learn how to interact with dynamic prompts parameters via on-policy reinforcement learning. during the inference phase, the attributes discriminator adjusts the prompt parameters based on the current generated results, and the model will generate the required text based on the current prompt constraints. since the language model already learns how to cooperate smoothly with the controller in the fine-tuning phase, the integrity of plm is improved during the generation process with the prefixes being the flexible global constraints. we have conducted sufficient experiments on topic control and emotion control tasks. the experimental results show that our method is significantly superior to the previous methods in terms of text generation fluency and quality, which proves that our method is very effective. our main contributions can be described as follows: ## related work the method to control the specific attributes or contents of the generated text has been widely studied @xcite @xcite . the most recent methods are built upon the large-scale pre-training language model (plm), which is based on transformers and used a large-scale corpus to learn copious language knowledge. in terms of nlg, plms can generate text with unprecedented quality. in general, an nlg system that is valuable in practical applications should be able to generate text that meets human expectations reliably. fine-tuning. in the above background, many pieces of research on controllable text generation based on plms have emerged. @xcite have studied the fine-tuning plms to complete the data-to-text task. the experiment shows that the effect of the model is better than that of the previous pipelined neural network model. reinforcement learning can also be used to control text generation. such methods fine-tune plm by taking whether constraints are met as rewards @xcite . @xcite training a scoring model to directly capture human preferences, then use this model to calculate rewards and train the generative model through reinforcement learning. prompt learning in order to make better use of the language understanding ability of plms, researchers have proposed a method called prompt learning, which allows plms to complete sentences according to the constructed prompt template without fine-tuning plms. the research of this method mainly focuses on how to build templates. @xcite propose the method of manually constructing templates. @xcite use an automatic search method to generate discrete prompts. after that, researchers proposed a continuous token template @xcite , this method is called prompt tuning. due to the serious impact of prompt design on its effectiveness, @xcite proposed to convert it into a learnable embedding layer. a similar method is prefix-tuning @xcite , which realizes controllable text generation by fine-tuning continuous parameters inserted in front of sentences. this method maintains the integrity of the plms and makes the survival text have a high fluency. however, the prefix parameters are fixed after training, which makes it necessary to train multiple different prefixes for different scenes, that is, this method is not plug-and-play. ## conclusion in this work, we propose prompt-ppc, which is a method to realize controllable text generation by dynamically adjusting prompts during model generation. we first propose a fine-tuning method to enable language models to understand dynamic prefix instructions, and in inference process, this method takes the prefix as a global constraint, provides a gradient through an external discriminator, and flexibly adjusts the prefix during the generation process to prompt plms to generate in the direction of the constraint. we have conducted experiments on topic control and emotion control tasks. the experimental results show that the fluency of the text generated by our method is very close to plms, and the diversity and control effect of the generated text is better than the previous methods. we hope that this work can broaden the thinking of prompt learning in the field of text generation. in the future, our method is expected to be applied to more complex and fine-grained control tasks.