Parameter-Efficient Detoxification with Contrastive Decoding (2024)

Abstract: The field of natural language generation has witnessed significant advancements in recent years, including the development of controllable text generation techniques. However, controlling the attributes of the generated text remains a challenge, especially when aiming to avoid undesirable behavior such as toxicity. In this work, we introduce Detoxification Generator (DETOXIGEN), an inference-time algorithm that steers the generation away from unwanted styles. DETOXIGEN is an ensemble of a pre-trained language model (generator) and a detoxifier. The detoxifier is trained intentionally on the toxic data representative of the undesirable attribute, encouraging it to generate text in that style exclusively. During the actual generation, we use the trained detoxifier to produce undesirable tokens for the generator to contrast against at each decoding step. This approach directly informs the generator to avoid generating tokens that the detoxifier considers highly likely. We evaluate DETOXIGEN on the commonly used REALTOXICITYPROMPTS benchmark (Gehman et al., 2020) with various language models as generators. We find that it significantly outperforms previous approaches in detoxification metrics while not compromising on the generation quality. Moreover, the detoxifier is obtained by soft prompt-tuning using the same backbone language model as the generator. Hence, DETOXIGEN requires only a tiny amount of extra weights from the virtual tokens of the detoxifier to be loaded into GPU memory while decoding, making it a promising lightweight, practical, and parameter-efficient detoxification strategy.

## introduction large language models (llms) have demonstrated remarkable promise in various generative tasks by first self-supervised pretraining on large text corpora and then finetuning with instruction data for * indicates corresponding authors. to address this challenge, a plethora of approaches have been proposed, which usually require full-model finetuning of the underlying language model to build the detoxifier @xcite @xcite . however, nowadays the largest llms typically contain more than 100 billion parameters, making such resource-intensive tuning less viable. this trend calls for more parameterefficient approaches. in this work, we propose detoxigen (figure 1 ), a parameter-efficient framework that leverages the frozen weights of the language model itself and only introduces a tiny portion of new model parameters to detoxify generation. 1 during training, we use prompt tuning @xcite to train a detoxifier exclusively on toxic data with the nexttoken prediction objective. the resulting detoxifier shares all the backbone model weights with the llm (i.e., the generator). during inference, we build on top of the contrastive decoding @xcite paradigm and employ the detoxifier to manipulate the output probability distribution of the llm for each generation step. intuitively, the generator avoids outputting tokens that the detoxifier considers highly probable. for example, in figure 1 the detoxifier considers the gender-biased word "his" very likely as the next token, helping the generator to score down the probability of that token. we evaluate our framework on the realtoxi-cityprompts dataset @xcite ) and find that it outperforms previous approaches on the standard benchmark metrics by a significant margin, indicating that the text generated by our model is both safer and of higher quality. we also conduct ablation studies and pair models of different sizes from the same model family (e.g., the llama-2 @xcite family). these studies show that pairing a generator with a detoxifier that shares the same backbone llm is indeed the best-performing configuration. our main contributions are: (1) performance: propose a detoxification framework that outperforms previous models by a large margin on commonly used detoxification benchmarks/metrics; (2) efficiency: we apply parameter-efficient learning to controllable text generation for detoxification. our model introduces the least amount of additional parameters (hence also requires less data to train) as compared to state-of-the-art models; (3) transferability: our detoxifier model only requires toxic data and does not require any contrastive (non-toxic) data, making our approach transferable thanks to the easier and more manageable data curation. ## discussion it would be ideal if a detoxifier could work out of the box (plug-and-play) and be readily applied to any llm generator, even with a different tokenizer. to achieve this, one can take a common subset of the vocabulary sets between the generator and the detoxifier, and only manipulate logits on this subset. we leave this as future work since the model families we investigate both already have diverse sizes. throughout the paper, we have been focusing on avoiding undesired attributes. however, we note that our framework can also be used to generate text with any desired style (which could be a risk if that desire style happens to be toxic). all we need to do is flip the sign of the probability distribution correction term ∆p in equation 2 and 3 as follows: in addition, our approach could be applied to more general positive and negative attributes, including but not limited to politeness @xcite , hate speech @xcite , and microagressions @xcite . in the case that we want to simultaneously control for multiple attributes, our framework is also compatible with mixed-batch inference @xcite , where soft prompts of different attributes can be conditioned on in a single batch without increasing latency. ## conclusion we propose detoxigen, a high-performing, parameter-efficient framework for detoxification during inference time. our method only introduces a small portion of new parameters to train a detoxifier model that manipulates the output probability distribution of a generative model. on a standard detoxification benchmark, our approach outperforms all existing models in terms of both toxicity and quality. as language models grow ever larger in size, our controllable generation method shows the promise of quickly adapting to any language model without requiring additional computational resources or significant data curation efforts.