Fine-Grained Controllable Text Generation Using Non-Residual Prompting (2022)

Abstract: The introduction of immensely large Causal Language Models (CLMs) has rejuvenated the interest in open-ended text generation. However, controlling the generative process for these Transformer-based models is at large an unsolved problem. Earlier work has explored either plug-and-play decoding strategies, or more powerful but blunt approaches such as prompting. There hence currently exists a trade-off between fine-grained control, and the capability for more expressive high-level instructions. To alleviate this trade-off, we propose an encoder-decoder architecture that enables intermediate text prompts at arbitrary time steps. We propose a resource-efficient method for converting a pre-trained CLM into this architecture, and demonstrate its potential on various experiments, including the novel task of contextualized word inclusion. Our method provides strong results on multiple experimental settings, proving itself to be both expressive and versatile.

## introduction a causal language model (clm) is a language model trained using a simple next-token prediction objective. current clms are typically based on the transformer architecture @xcite , which has resulted in unprecedented text generation capabilities @xcite . even so, the generation process of a clm is difficult to control, as one is forced to gradually decode the next-step prediction one token at a time. this inhibits the applicability of clms when one intends for the generated text to fulfill certain criteria, and not only be a linguistically sound continuation in a given context. being able to control the text generation process is crucial for many real-world applications. as a straightforward example, we may want to control the generated text to counter the many biases that modern clms have been shown to possess @xcite . however, most applications require a greater degree of control, as one often wishes to steer the text generation in a specific direction, such as generating a story to a given plot @xcite @xcite , or sticking to a certain topic @xcite . some areas require stringent and fine-grained control, as the many data-to-text tasks @xcite @xcite , which necessitates that the generated text mediates very specific information and facts. due to this apparent need for controllable text generation, recent work (see section 2.1) has explored different methods to steer and constrain the generation process of a clm. there are mainly two lines of research in this area. the more traditional approach focuses on fine-grained control and how to steer the generation process at arbitrary points, while still adhering to the current context. this is often achieved by independently modifying the predicted vocabulary distribution at each decoding step. however, this decouples the clm from the control method, prohibiting the clm's ability to plan accordingly and thus severely limits the type of control that can be formulated. the second approach instead opts for more expressive and high-level control, letting the clm itself interpret and incorporate the instruction into the text generation. this is often done via either a fine-tuning objective or, as is currently common, by formulating the instruction as a textual context (referred to as prompting). although expressive, these approaches are less effective than the previous ones in controlling generation at specific points. this is due to the prompt's influence being negatively correlated with the distance from the prompt to the next predicted token @xcite , making prompting difficult for nonadjacent text. in an attempt to bridge the gap between finegrained control and the expressiveness of prompts, we propose an architecture that permits longdistance and independent prompting throughout the generation process. this architecture has an encoder-decoder setup, where the encoder influences the decoder via a novel non-residual attention schema. along with theoretical arguments for the benefits of this architecture, we provide a resource-efficient self-supervised method for converting a pre-trained clm into this setup. in addition to evaluating on the original com-mongen dataset @xcite , we propose a new contextualized version of commongen, called contextualized commongen (c 2 gen) and evaluate relevant methods on it. this new dataset extends the task to generating a sentence which includes a given set of words, while simultaneously adhering to a given context. we find that no previous solution is capable of handling this task, either barely including 50% of the target words, or not generating text of satisfactory quality. ## model architecture we propose to steer a clm's generative direction by introducing a separate "encoder" for prompt instructions, which we refer to as the prompt model. the prompt model interprets textual prompts and produces positional invariant key-values, which the clm can attend to via the novel non-residual attention schema (section 3.1). the positional invariance ensures that the instruction is equally applicable at any time step, and is achieved by an additional shift of its key-values (section 3.2). ## training procedure given a pre-trained clm, we propose to train an accompanying prompt model via four distinct phases, foot_3 as demonstrated in figure 2 . this includes an initialization phase, two pre-training phases, and one optional fine-tuning phase. the weights of the clm are never updated in any of the training phases. as popularized by @xcite , all training, independent of task, is formulated within the framework of teacher-forced causal language modeling @xcite . the goal is to maximize the likelihood of generating text s given prompt p , in accordance with equation 1. ## word inclusion experiments we separate word inclusion into two different settings. in the first, the model is tasked to generate exactly 32 tokens. requiring the model to both satisfy the word inclusion objective, and continue generating contextually relevant text. this allows methods that do not grant sentence level control to participate, such as pplm and keyword2text. in the second setting, the model is only tasked to create a single sentence, elevating the requirement of continued text generation. this setting is suitable for methods specifically trained towards creating a single common sense sentence, such as kg-bart and pointer. for both of these settings, we run experiments on both commongen and c 2 gen. since experiments on the contextualized c 2 gen require the model to adhere to a context regardless of whether the objective is to generate a single sentence or a free text, kg-bart and pointer are excluded from these experiments all together. ## sentence length experiments the inclusion of sentence length in the pretraining objective (section 4.2), gives an additional level of generative control over the linguistic style. as demonstrated in lp lg target words generated sentence 6 6 drink, sit, table, wine the wine-drinkers sit on the table. 12 11 drink, sit, table, wine the guests sit at a table and drink wine or beer. 18 16 drink, sit, table, wine the guests sit at a table and drink wine, while the hostess sits on the floor. 8 8 amazing, trump, politics, victory the trump victory was an amazing victory in politics. 14 14 amazing, trump, politics, victory the trump victory was an "amazing" and "incredible" victory for american politics, he said. 20 25 amazing, trump, politics, victory the trump victory in the 2016 presidential election was an amazing victory for american politics as a whole, but it wasn't just about donald j. we note that the model tends to prioritize textual quality over strictly sticking to the exact number of words. to measure this discrepancy, we generate sentences for all commongen validation samples for different prompted sentence lengths. figure 4 shows the results from this experiment, displaying the expected offset for different prompted sentence lengths. the mean offset is always above 0 and below 1, meaning the clm can be expected to generate a slightly longer sentence than intended. the standard deviation increases both as the prompted length approaches long, and short sentences. this matches the sentence distribution of the pretraining dataset, as demonstrated in ## discussion and future work we opted to demonstrate our architecture's capabilities on the task of word inclusion, since quantitative comparisons on this task are relatively straight-forward, compared to most other openended text generation tasks. while experimental results indicate the versatility of our approach, it is important to note that the method conceptually generalizes to a much wider range of tasks. our non-residual architecture enables the use of prompt instructions at arbitrary time steps, but is not limited to word inclusion. we hence encourage future work to pursue the incorporation of multi-task prompt learning, as being able to apply flexible prompts with precision would be a big step forward in the many areas striving to use clms. indeed, we consider the ability to control the text generation process while considering context crucial for any tool intended for human editors. admittedly, our training method for realizing our encoder-decoder architecture has largely been dictated by a lack of resources. we conceptually prefer the more straight-forward training approach of training the prompt model directly on long context data, and removing the positional invariant transformation. future work could thus increase computational resources and investigate the possibility of different positional encoding schemes. finally, we stress that nothing in our approach has focused explicitly on common sense. it is hence expected that methods that do, like kg-bart, perform better on this metric. future work could thus investigate the use of a prompt model to control a clm fine-tuned towards common sense, or fine-tune a prompt model using common sense data. results on commongen and c 2 gen dataset still leave ample room for improvements. ## 9 conclusion this paper has introduced the concept of nonresidual attention and demonstrated how it can be used to control a generative text model. additionally, our work pinpoints the lack of open-ended controllable text generation tasks that require the model to also account for a given context. we set out to remedy this by introducing the humanly created c 2 gen dataset, introducing the task of contextualized word inclusion. experimental results on c 2 gen and com-mongen, clearly demonstrates that using a nonresidual prompt model increases generative control over a clm. compared to other methods, our approach stands out as the most versatile, consistently performing well across all tested situations.