Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models (2022)

Abstract: We explore the idea of compressing the prompts used to condition language models, and show that compressed prompts can retain a substantive amount of information about the original prompt. For severely compressed prompts, while fine-grained information is lost, abstract information and general sentiments can be retained with surprisingly few parameters, which can be useful in the context of decode-time algorithms for controllability and toxicity reduction. We find that some complex prompts can be effectively compressed into a single token to guide generation. We also show that compressed prompts are largely compositional, and can be constructed such that they can be used to control independent aspects of generated text.

## introduction language models (lms), such as gpt-2 @xcite , bert @xcite , t5 @xcite , or @xcite , exhibit a remarkable ability to capture patterns of grammar, vocabulary, cultural knowledge, and conversational rhythms present in natural language. formally, a lm is a conditional distribution over tokens p(x t |x 1 , • • • , x t-1 ), with each token x @xmath0. throughout this paper, we will refer to x @xmath1, • • • , x t-1 as the prompt. this paper explores prompt compression: the idea that the text x h used to condition a lm can be approximately represented by a much smaller set of carefully chosen weights, using the framework of soft prompts @xcite . we begin by establishing some basic properties of compressed prompts, and importantly show that while highly lm lm kl hard prompt soft prompt compressed prompts lose fine-grained information about the prompt, they can retain general, abstract information. this motivates our central application: to use such compressed prompts in a bayesian attribute framework to steer text generation, with specific application to toxicity reduction. to motivate this more deeply, we briefly sketch how compressed prompts can be used in toxicity reduction. efforts to reduce toxicity and bias generally follow one of two strategies: the first is to train or fine-tune lms on carefully curated data, either tagging or labelling it in special ways @xcite or using data known to be "clean". the second is to "steer" the generation of token probabilities away from toxic generations @xcite , and towards text with known, desirable properties. following previous work, we steer lm probabilities by using a bayesian attribute classifier framework that involves scoring candidate tokens with different experts. as an independent contribution, we explore the idea of simply using conditioning text to construct such experts by leveraging the few-shot modeling abilities of lms @xcite : given a few examples of text containing a pattern of interest, language models are capable of "analyzing" such examples and assign high probability to subsequent text exhibiting the same pattern. thus, in the same way that language model can, for example, clas-sify the sentiment of a tweet, we use lms to analyze the toxicity of candidate generations in realtime. our method can be considered an exemplarbased method of defining experts that capture desirable and undesirable attributes of generated text. we term this technique contrastive contexts, and note that it reduces the problem of creating experts to one of prompt engineering @xcite . however, our conditioning contexts are quite large, which motivated this work. we use prompt compression to mimic an uncompressed prompt (hereafter referred to as "hard" prompt) as closely as possible, thereby saving both computation and space in the context window. our results demonstrate that this can be very effective, and, in a very surprising finding, that complex prompts can be reduced to a single token and still be useful for toxicity reduction, often with better fluency compared to hard prompts. the contributions of this paper are three-fold: first, we introduce and formalize the idea of prompt compression; second, we introduce and formalize the method of contrastive contexts in the bayesian attribute framework; third, we experimentally evaluate our methods, and refine the technique based on various empirical observations, and contribute a careful study of effectiveness as model size varies. our code is available online. 1 ## background and related work to the best of our knowledge, this is the first work to directly explore prompt compression. however, our work is based on the original soft prompt ideas of @xcite . it is also somewhat related to distillation, where one model is trained to mimic another by matching logits @xcite . usually, lms take text as input, which is then tokenized into discrete tokens by a tokenizer. each token is then mapped to a learned embedding, which is used as input to a transformer @xcite . the idea of soft prompts @xcite is to bypass the need to use discrete tokens with pre-trained embeddings and instead directly learn a series of embeddings via backpropagation. these learned embeddings are then fed directly to the transformer and do not need to correspond to any actual language tokens. as the centerpiece application of prompt compression, we explore generative controllability @xcite and toxicity reduction in language models. our method is most closely related to decodetime algorithms, such as gedi @xcite , which uses bayes' rule and discriminative models to steer the generation towards a certain attribute; and pplm @xcite , which uses an estimated gradient with respect to the desired attribute to steer the lm's internal representation at generation time. other methods are based on fine-tuning language models with the classical language modeling objective to steer generation. dexperts @xcite combines experts and anti-experts in a product of experts model to reduce toxicity of lms. additionally, reinforcement learning approaches show strong performance at steering language models @xcite . by providing rewards, methods such as ppo @xcite and quark @xcite represent the current best performance at reducing lm toxicity while maintaining fluency. these methods, however, require a predetermined reward function, which may or may not be feasible depending on the context. ## prompt compression here, we introduce and explore the idea of prompt compression, whereby the parameters of a soft prompt @xcite are trained to mimic a fixed hard prompt as closely as possible. the intuition of our idea is simple: conditioning a lm on a hard prompt x h induces a distribution p(x t , • • • , x t+k |x h ) over all possible sub-sequent sequences of tokens x t , • • • , x t+k for all k. to simplify notation, let x t:@xmath2, • • • , x t+k . the schematic of the idea is shown in fig. 1 . formally, a soft prompt is a block of weights θ n that is prepended to the embeddings of the tokenized sequence x t:k , and which is then fed through the transformer layers of the language model. the soft prompt induces a modified distribution over x t:k , which we represent as q(x t:k |θ n ). here, n is the number of tokens in the soft prompt (which do not necessarily correspond to natural language tokens). to compress prompt x h , we train the soft prompt weights to minimize the following objective: ## experiment set #1: establishing basic properties of compressed prompts we begin by exploring the properties of compressed prompts. first, we show that conditioning on a hard prompt and its compressed prompt generate qualitatively similar generations, although this equivalence degrades as the prompt is compressed more and more. second, we qualitatively explore what happens to fine-grained information as a prompt is compressed more and more. ## the bayesian attribute classifier framework as an application of compressed prompts, we now turn our attention to toxicity reduction and controllability. following previous work @xcite we adopt the bayesian attribute classifier framework for decode-time controllability. our goal is to generate text that exhibits some attribute a; by conditioning generations on this attribute and using bayes law, we arrive at there are multiple ways to construct the attribute classifier p(a|x h , x t ). if the desired attribute a is, for example, "does not contain profanity", then the attribute classifier could simply scan x h for words on a blacklist and output p(a|x h , x t ) = 1 if none of the words are present. more sophisticated approaches are possible; our approach centers on the careful construction of this classifier. our work is most similar technically to the gedi framework @xcite , which uses two language models to construct the classifier in a contrastive manner. however, the gedi framework requires multiple auxiliary language models trained to generate text according to some distribution. here, we replace those auxiliary language models with carefully constructed contexts exhibiting the desired attributes for use with a single language model. ## experiment set #2: application to toxicity reduction to empirically assess prompt compression in the context of toxicity reduction, we follow the experimental protocol outlined in the realtoxici-typrompts (rtp) paper @xcite . the rtp paper contributes both a dataset and a variety of metrics for assessing toxicity; we briefly summarize those here. also following the rtp paper, all toxicity measurements are done with the perspectiveapi (jigsaw and the google counter abuse team, 2015), an imperfect @xcite but standard tool for assessing toxicity along a variety of dimensions. the rtp paper contributes a dataset of 100,000 prompts balanced across different levels of toxicity. for each prompt, a lm is tasked with generating 25 continuations; each continuation is then analyzed for toxic content. there are two primary metrics of interest. the first is the expected maximum toxicity, where the max toxicity is taken over the 25 generations, and second is the average toxicity. ## conclusions and future work we have explored the idea of prompt compression, establishing basic properties of the method and then examining an extended application to controllability and toxicity reduction. based on our experiments, we conclude that prompts can be significantly compressed and still retain some useful information. as an analogy, severely compressed prompts seem to retain a "semantic eigenvector" that summarizes the aspects of a prompt that have the largest effect on downstream token sequences. this suggests that representing information as basic tokenized sentences is inefficient, and that more general prompt compression strategies may be possible (for example, by training a prompt-compressing deep neural network). we see that s m l xl steering model s m l xl base model 0.32 0.38 0.39 0.4 0.33 0.38 0.4 0.41 0.33 0.39 0.41 0.43 0.34 0.39 0.4 0.42 hard prompts s m l xl s m l xl 0.31 0.35 0.37 0.38 0.32 0.36 0.38 0.4 0.33 0.38 0.39 0.41 0.33 0.38 0.39 0.42