Prompt-Based Length Controlled Generation with Multiple Control Types (2024)

Abstract: Large language models (LLMs) have attracted great attention given their strong performance on a wide range of NLP tasks. In practice, users often expect generated texts to fall within a specific length range, making length controlled generation an important topic, especially for GPT-style models. Existing length control methods mostly focus on a simple control type of “equal to” a target length. Different from them, we propose a prompt-based method to achieve length controlled generation under different control types with high accuracy. In particular, we adopt reinforcement learning (RL) and sample filtering with the reward signal given by rule-based reward models, which enhances the length control ability of models by rewarding outputs that follow certain control instructions. In addition, we introduce a standard prompt extractor to parse arbitrary users’ input into standard control instructions. Experiments show that our method significantly improves the accuracy of prompt-based length control on popular summarization datasets like CNNDM and NYT under multiple control types. Moreover, both the standard prompt extractor and RL-tuned model show strong generalization to unseen control prompt templates.

## introduction for recent popular gpt-style models like chat-gpt and gpt-4 @xcite @xcite , various studies have been conducted on them, and the inference efficiency and computational cost often draw concerns from the community @xcite @xcite . since its generation is in an autoregressive manner, the inference cost increases continually with the growing of decoding steps. meanwhile, users of llms usually have an expected length of generated texts, *work is done as a postdoctoral research fellow at noah's ark @xcite no matter for writing an essay or summary, knowledge qa or dialogue generation @xcite @xcite @xcite . both of these two facts require the length of generation in gpt-style models can be effectively controlled. for pretrained language models (plms), the most widely applied technique for length control is prompt-based fine-tuning @xcite @xcite . taking an example of length-controlled summarization (lcs), we can prepend a prompt "summarize with length l i :" to the article to be summarized in training, where l i is the number of words of reference summary. however, this process is usually performed in supervised fine-tuning (sft), where the length control ability has to compromise with the goodness of downstream tasks. for large-scale models like gpt-3, the length controlled ability can be somewhat activated by incontext learning without any fine-tuning @xcite @xcite . however, it relies on the size and power of the pre-trained foundation models to achieve good performance. for other methods like rlhf @xcite @xcite , it is expensive to manually label whether the generated length meets the requirement given in instruction prompts. meanwhile, automatic reward labels can not be straight forwardly obtained, as the control instructions can be arbitrarily integrated into user utterances. generally, there are many other length control methods such as golc, lenatten and laam @xcite @xcite @xcite . however, these methods are not particularly designed for plms, thus architecturespecific designs on training mechanisms are usually needed. moreover, they often focus on the setting of equalling to a certain length, generally not adapt to other control types such as greater/smaller than a value, or between two values, etc. meanwhile, they can not handle diverse expressions of control instructions from users. therefore, how to effectively connect diverse control instructions from users to the length of generated text for plms is still an open question. in this paper, we introduce a novel method that applies prompt-based fine-tuning with reinforcement learning to improve the performance of length controlled generation, which is capable to handle multiple types of length control at the same time. our main contributions are: reinforcement learning (rl) @xcite has been widely applied to improve text generation performance, including summarization @xcite , question generation @xcite , and dialogue generation @xcite @xcite . in general, we can consider the generative model as the policy network and optimize its parameters for achieving higher reward from the environment @xcite . human feedback is one of the most known strategies to get the reward, which is shown to be more effective than using some automatic metrics, such as rouge scores in text generation @xcite @xcite . existing study @xcite also shows that rl techniques are generally better than supervised methods at aligning language models to human preferences. it is known that reinforcement learning from human feedback (rlhf) plays a key role in the success of autoregressive llms like instructgpt @xcite , which uses human feedbacks to train a reward model for ppo @xcite . ## method we aims to develop the length-controlled generation methods for gpt-style plms, especially for the cases with multiple control types. we first introduce the whole architecture in section 3.1 and then discuss each component of it. ## conclusion we proposes a method for improving the length control ability of gpt-style plms under multiple control types, especially for the domain of text summarization. the standard prompt extractor and rule-based reward model are introduced to provide an accurate control signal for both fine-tuning and inference. we apply a modified ppo algorithm for enhancing the length controlled generation. in the inference, sample filtering is further introduced for selecting a generated sample that follows the instruction. the method is proved to be effective for three sizes of gpt-2 models on both cnndm and nyt summarization datasets. compared to the baseline using prompt-based strategies on gpts, our method further achieves a significant improvement in terms of control accuracy. moreover, it can process diverse length-control prompts with strong generalization ability to new prompt templates, and can naturally adapt to most llms for improving user experience.