PREADD : Prefix-Adaptive Decoding for Controlled Text Generation (2023)

Abstract: We propose Prefix-Adaptive Decoding (PREADD), a flexible method for controlled text generation. Unlike existing methods that use auxiliary expert models to control for attributes, PREADD does not require an external model, instead relying on linearly combining output logits from multiple prompts. Specifically, PREADD contrasts the output logits generated using a raw prompt against those generated using a prefix-prepended prompt, enabling both positive and negative control with respect to any attribute encapsulated by the prefix. We evaluate PREADD on three tasks—toxic output mitigation, gender bias reduction, and sentiment control—and find that PREADD outperforms not only prompting baselines, but also an auxiliary-expert control method, by 12% or more in relative gain on our main metrics for each task.

## introduction the dramatic rise in applications relying on language models has led to increased interest in methods for controlling their generations based on desired constraints. for example, it is desirable to prevent models from generating toxic or harmful text, foot_0 as they are often prone to doing @xcite , especially in the presence of toxic prompts. to this end, prior work has proposed many viable control schemes, ranging from prompting with instructions to specify a constraint @xcite , to using an auxiliary expert model to guide generation @xcite . however, for important practical tasks such as toxic output mitigation and gender bias reduction requiring control against an undesired attribute, prompting-only methods may struggle @xcite , as we observe in our own experiments (section 4). in failure cases, it is unclear how to adjust control strength when relying solely on prompting. approaches using auxiliary models may be advantageous in this respect, but auxiliary models impose an additional burden in practice, typically requiring training data. additionally, prompting approaches may naturally improve as the base language model improves, which is not necessarily the case when relying on an auxiliary model for control. in this work, we propose prefix-adaptive decoding (preadd), a prompting-only control scheme that enables adjusting control strength (figure 1 ). preadd operates by contrasting the token logits at each step of generation when using either (1) a prefix-prepended version of a prompt, or (2) the raw unmodified prompt. the difference between logit distributions can then be amplified or negated to vary the control strength, as required for the task. we evaluate preadd on toxic output mitigation and gender bias reduction, two tasks which require "negative" control against an undesirable attribute. we believe preadd offers the largest advantage over traditional prompting approaches in such settings. on these two tasks, preadd significantly improves over prompting-only baselines and also an auxiliary-model control method by 12% or more in relative improvement on our main metrics for each task. meanwhile, preadd still maintains strong performance on "positive" control tasks such as sentiment control. all code is available at https://github.com/ jonnypei/acl23-preadd . ## related work prior works have attempted to control language model outputs through a variety of methods. prompting. prompting approaches have become the manager is a complete for the original tokens x1:i, as well as p (xi+1 | r 1:k , x1:i) for x1:i prepended with an additional toxicity-encouraging prefix r 1:k . xi+1 is then sampled proportional to p (xi+1 | r 1:k , x1:i) α p (xi+1 | x1:i) 1-α , with α set to a negative value to control against the toxicity encouraged by r 1:k . meanwhile, probabilities of unrelated tokens (e.g., "and") are kept relatively unchanged. auxiliary models. meanwhile, control schemes using auxiliary models for a desired attribute typically provide avenues to adjust the strength of control as needed. while some methods require deeper access to the base language model, such as gradients @xcite , others require only the output token logits at each decoding step @xcite @xcite . however, auxiliary models may require additional training data to learn the desired attribute, unlike prompting methods such as preadd. methods such as finetuning or reinforcement learning from human feedback @xcite may also use additional training data to modify the model distribution at training time rather than inference time. ## prefix-adaptive decoding we now motivate and develop our method. suppose we want to control the output of a language model g, which generates tokens x 1 . . . x n left-to-right by modeling p (x i+1 | x 1:i ). one possible method is prompting: we prepend a prefix r 1:k to x 1:i , modeling x i+1 according to p (x i+1 | r 1:k , x 1:i ). while prompting is lightweight and flexible, we may wish to adjust the control strength when working with highly complex constraints. some constraints may also be difficult to express effectively as a prompt: for example, simply stating that the model should not generate text of a particular attribute is often ineffective (section 4). we thus develop our method for controlled generation, prefix-adaptive decoding (preadd, figure 1 ), to allow varying the strength of control during prompting. one can view prompting as modulating the log-probabilities log p (x i+1 | x 1:i ) by adding the difference in log-probabilities: intuitively, adding d increases the likelihood of tokens relevant to the prompt, while leaving that of unrelated tokens (e.g., stopwords) largely unchanged. applying a multiplier α to d may therefore enable us to vary control strength while preserving fluency. thus preadd models the logit of x i+1 as: converting back to normal probability space and re-expanding d, we obtain the final probability for x i+1 according to preadd, proportional to: preadd strictly generalizes prompting; the latter is equivalent to α = 1. unlike prompting, preadd can control more strongly for the constraint expressed in the prompt by using larger values of α, or can provide negative control against the constraint expressed in the prompt by using negative values of α. we explore both cases in our experiments (section 4) ## experiments we test preadd on three controlled text generation tasks: (1) toxicity mitigation, (2) gender bias reduction, and (3) sentiment control. ## discussion in this work, we have proposed preadd, a prompting-based method for controlled generation. unlike typical prompting approaches, preadd can adjust the degree of control exerted by the prompt by contrasting the output logit distributions for two different prompts, allowing for flexible control strength similar to auxiliary-model-based control methods without requiring training data. in our experiments, preadd outperforms both simple prompting baselines and an auxiliary model method on three different tasks: toxic output mitigation, gender bias reduction, and sentiment control. in principle, preadd is highly flexible and can be applied to a wide range of other tasks as well. for instance, one could use preadd to increase control strength to satisfy more difficult, complex constraints such as faithfulness to a story outline @xcite , or one could extend preadd to contrast more than two prompts at a time to satisfy multiple simultaneous constraints.