DE xperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts (2021)

Abstract: Despite recent advances in natural language generation, it remains challenging to control attributes of generated text. We propose DExperts: Decoding-time Experts, a decoding-time method for controlled text generation that combines a pretrained language model with “expert” LMs and/or “anti-expert” LMs in a product of experts. Intuitively, under the ensemble, tokens only get high probability if they are considered likely by the experts, and unlikely by the anti-experts. We apply DExperts to language detoxification and sentiment-controlled generation, where we outperform existing controllable generation methods on both automatic and human evaluations. Moreover, because DExperts operates only on the output of the pretrained LM, it is effective with (anti-)experts of smaller size, including when operating on GPT-3. Our work highlights the promise of tuning small LMs on text with (un)desirable attributes for efficient decoding-time steering.

## introduction controlling the output of pretrained language models (lms) is crucial for achieving useful and safe language generation applications, such as nonoffensive sentence completion or friendly conversation generation @xcite @xcite . for example, a safe completion to the prompt "when she rejected his advance, he grabbed..." requires avoiding word choices that could lead to continuations with gender-based violence (e.g., "her"; figure 1 ). without such steering, these language models risk generating mindless and offensive content @xcite ) which hinders their safe deployment @xcite . importantly, as the scale of pretrained lms increases (e.g., 175b and 1.6t parameters; @xcite figure 1 : illustration of dexperts, where a toxic lm acts as an "anti-expert" and a non-toxic lm acts as an "expert". in this toy example, given the prompt, "when she rejected his advance, he grabbed," the toxic lm assigns greater weight to "her" than "his", expressing subtle signals of toxicity that can be leveraged for effective attribute control. the difference in logits z `´z óutput by the expert and anti-expert represents the perturbations to make to the logits z of the pretrained "base" lm. we propose dexperts, 1 a decoding-time method for controlled text generation based on a product of experts @xcite . our method combines an out-of-the-box pretrained ("base") lm with "expert" lms and/or "anti-expert" lms, which model text with desirable and undesirable attributes, respectively. by generatively modeling text with particular attributes and directly combining the output distributions from each lm, dexperts leverages subtle signals expressible by language models for effective attribute control, without sacrificing generation fluency or diversity. moreover, because it operates only on the output of the base lm, dexperts can steer with (anti-)experts of smaller size, even in cases where we do not have full access to the base model (e.g., gpt-3 through an api). we first apply dexperts to the task of language detoxification ( §3), by finetuning an expert and an anti-expert on public comments that are humanannotated for toxicity. our experimental results show that dexperts can successfully avoid toxicity in language generation while preserving output fluency, outperforming existing detoxification methods on both automatic and human evaluations. moreover, we find that dexperts continues to outperform baselines when employing only an antiexpert and re-using the base model as the expert, making it one of the only methods that can avoid toxicity without annotated examples of non-toxic content. in analysis, we also show that our method successfully avoids toxic degeneration while using just "650 toxic comments, opening avenues for easily customizable anti-experts. we then showcase the generalizability of dex-perts by tackling the task of controlling the sentiment of lms' output ( §4). to this end, we combine a pretrained lm with (anti-)experts modeling positive and negative sentiment. as with language detoxification, dexperts outperforms existing sentiment steering methods on both automatic and human evaluations. additionally, we show our method is especially effective in the adversarial setting of steering negative prompts toward positive continuations, and vice versa. finally, we demonstrate a preliminary proof-of-concept using dexperts for stylistic rewriting ( §5). our work demonstrates the effectiveness of tuning small lms on text with desirable and undesirable properties for efficient and effective steering of larger pretrained lms, and highlights the promise of decoding-time methods for controlled language generation. ## experts and anti-experts for controlled generation given input text as a prompt, the task of controlled text generation is to generate a continuation that flows naturally from the prompt while having the desired attribute (e.g., positive sentiment) but not an undesired one (e.g., toxicity). given a prompt x ăt , the language model computes the logits for the tth token, denoted z t p r |v| , where v is the vocabulary. a probability distribution over the vocabulary is obtained by normalizing and exponentiating z t : and the next token is generated by sampling x t " p px t | x ăt q. ## toxicity avoidance given that large pretrained lms are at risk of producing toxic content @xcite , steering away from toxic "degeneration" is crucial for their safe deployment. our approach uses an anti-expert that models overt toxicity, as well as an expert that is finetuned on nontoxic data from the same domain. note that while obtaining an lm that is truly free from social biases is impossible @xcite , the "non-toxic" expert serves the purpose of modeling the same domain of comments as the toxic anti-expert, providing more effective contrast. nonetheless, we provide an ablation using only a toxic anti-expert and show that it remains effective above all previous baselines. ## sentiment-controlled generation as a second application we consider the wellstudied task of controlling the polarity of text's sentiment (e.g., @xcite , steering towards either positive or negative sentiment. ## stylistic rewriting with dexperts as a preliminary exploration, we go beyond generating text continuations to apply dexperts to stylistic rewriting, i.e., rewriting a sentence in a target style while preserving as much content as possible. we replace the base model with a pretrained autoencoder, bart @xcite , and use gpt-2 large sentiment (anti-)experts from §4 for steering. at each time step, the autoencoder base model conditions on both the input sequence and the generation-so-far, whereas the (anti-)experts condition on only the latter. as a proof of concept, we show some examples of input/output from this system in input ñ output examples i love cats and seeing them play with yarn. oatmilk is tasty and good for the environment. this exploration suggests that more innovation is required to apply dexperts to stylistic rewriting, but it is a promising direction. we anticipate future work on the subject. ## related work the task of controlling the output of a language generation model has been widely studied by previous work (for a review, see @xcite . prior to using pretrained lms as a backbone, most work used custom neural models trained for their respective downstream generation tasks, including emotion-aware text generation @xcite , attribute-aware product review generation @xcite , and friendly or empathetic dialogue response generation @xcite . since pretrained lms have shown impressive text generation ability @xcite , two directions have emerged to control their language generation: training approaches and decoding-time approaches. training approaches include finetuning the pretrained lms on datasets that contain the desired attributes @xcite as well as creating a class-conditioned pretrained lm trained on text with specific attributes control code prefixes @xcite . in contrast to our method, such approaches can only steer towards desired text attributes, they cannot steer away from them. additionally, training approaches require significant computational resources, which may no longer be feasible with the size of more recent pretrained lms @xcite . decoding-time methods, a more lightweight approach, have been used controlling the attributes of generated text, as well as for improving its quality @xcite @xcite . pplm @xcite is a steering method that updates a pretrained model's hidden representations according to the gradient of a classifier with respect to the desired class. unfortunately, this approach is computationally expensive, as shown in this and previous work @xcite . contemporaneous with our work, fudge @xcite trains classifiers on partial sequences to predict whether an attribute will be satisfied in the future, and uses bayesian factorization to obtain the attribute-conditioned probability distribution. gedi @xcite uses bayes' rule similarly, but computes classification probabilities using the output of class-conditioned lms rather than directly training a classifier. in contrast, our experiments show that directly ensembling lms' probabilities as opposed to using them for estimating class probabilities is more effective at steering text generation. ## conclusion we present dexperts, a method for controlled text generation that reweights the predictions of language models based on expert (and anti-expert) opinions. in experiments for two different tasks, detoxification and sentiment control, we show that our method is able to effectively steer the language model towards the desired generations, while preserving the fluency and diversity of generated text. as applications built on language models become ubiquitous, dexperts demonstrates promise in steering these models toward safe and user-friendly generations.