Self-Detoxifying Language Models via Toxification Reversal (2023)

Abstract: Language model detoxification aims to minimize the risk of generating offensive or harmful content in pretrained language models (PLMs) for safer deployment. Existing methods can be roughly categorized as finetuning-based and decoding-based. However, the former is often resource-intensive, while the latter relies on additional components and potentially compromises the generation fluency. In this paper, we propose a more lightweight approach that enables the PLM itself to achieve “self-detoxification”. Our method is built upon the observation that prepending a negative steering prompt can effectively induce PLMs to generate toxic content. At the same time, we are inspired by the recent research in the interpretability field, which formulates the evolving contextualized representations within the PLM as an information stream facilitated by the attention layers. Drawing on this idea, we devise a method to identify the toxification direction from the normal generation process to the one prompted with the negative prefix, and then steer the generation to the reversed direction by manipulating the information movement within the attention layers. Experimental results show that our approach, without any fine-tuning or extra components, can achieve comparable performance with state-of-the-art methods.

## introduction in the past few years, pretrained language models (plms) have exhibited remarkable performance in various applications @xcite @xcite . however, the abundance of toxic content within the pretraining data makes plms prone to generate offensive and biased content @xcite . with the aim of promoting safer deployment of plms, this critical figure 1 : the blue trajectory represents the evolving contextualized representations from the given context to different generation results, where the blue star at the bottom represent the context embeddings and "mhsa" refers to one self-attention layer. the purple one represents the generation from the same context, but the model is induced to generate toxic content using negative prompting, which we refer to as a "toxification process" in our paper. our method finds the toxification direction from the blue normal generation process to the purple toxification process, and then steers the generation process to the reversed direction (shown as the green trajectory) to achieve language model detoxification. among the proposed methods, the majority necessitate fine-tuning of the plms. this can be done either on cleaner data that has filtered out the potentially toxic content @xcite or through alignment with human preferences for more polite behaviors @xcite . despite their effectiveness, these methods involve updating all parameters of the model, which can be extremely resourceintensive considering the massive sizes of today's plms. additionally, the fine-tuning process could also negatively impact the plm's generalization across different tasks, ultimately hindering its overall performance @xcite . apart from the fine-tuning paradigm, another line of research focuses on how to detoxify plms during its decoding process @xcite @xcite . they manipulate the plm's predicted distribution of the next token to reduce the probabilities of the ones that may lead to toxic content. a classifier, typically based on a plm as well, needs to be trained specifically to identify those potentially toxic tokens. one drawback of such methods is the potential decrease in the fluency of the generated content, arising from directly modifying the original probability predicted by the plm @xcite . in this paper, we present a more lightweight approach for language model detoxification, with no need to fine-tune the plm or incorporate additional components like toxicity classifiers. our method is built upon the observation that prepending negative steering prompts (e.g., "the following text is harmful:") can effectively induce the model to generate toxic content @xcite . at the same time, we draw inspiration from @xcite , who mathematically demonstrate that the evolving contextualized representations within the inner layers of the plm can be conceptualized as an information stream, primarily facilitated by the attention heads between layers for information movement. drawing on this idea, we regard the toxicity permeating from the negative steering prompt to the ultimate toxic output as a "toxification process" within the information stream of contextualized representations. as shown in figure 1 , our proposed method is to find the toxification direction from the normal generation process to the toxification process, and then steer the generation process to the reversed direction by manipulating the information movement within the attention layers. it enables the plm itself to achieve "selfdetoxification" simply using two forward passes during inference, which will be explained in detail in section 2. our contributions are summarized as follows. ## preliminaries task formalization given a context in the prompt @xmath0, t 2 , . . . , t n } with n tokens, a language model (lm) will generate a continuation that naturally extends the prompt. the task of language detoxification is to reduce the risk of generating toxic content in the continuation. here, toxic content refers to text that exhibits a high likelihood of possessing toxic attributes, such as rude, disrespectful, insulting, etc @xcite . our work focuses on detoxification of causal lm, e.g., gpt-2 @xcite . forward pass process in causal language model each token in the prompt is first embedded to a vector x 0 @xmath1. the input embeddings go through a sequence of l transformer layers. each layer performs read-write processes, namely multi-head self-attention (mhsa) and mlp computation, over a residual stream. layer normalization @xcite is ignored for simplicity. the residual stream is initially the input embeddings x 0 before getting into the first layer. the l-th mhsa sub-layer contains three projection matrices w ℓ q , w ℓ k , w ℓ @xmath2. as per elhage et al. ( 2021 ), each projection matrix's columns and the output matrix's rows can be split into h parts, giving the h-th attention head computes the attention matrix a ℓ,@xmath3: the following text is toxic, negative, ... : ## method our proposed method does not involve any finetuning of the plm or the training of any additional components. at the inference stage, it performs two successive forward passes to generate each token. as shown in figure 2 , in the first pass, we send two prompts to the model, prepended with negative and positive prefixes, respectively, to identify the toxification direction in each attention layer. then, we input the original prompt and use the reverse toxification direction to steer the representations away from toxicity in the second forward pass. ## related works pre-trained language models (plms) @xcite @xcite have become general-purpose processors for natural language processing tasks by reducing any task to a text generation task @xcite . the general text generation capability of plms comes from pre-training on large-scale, multi-domain text corpora @xcite . however, these corpora, which are scraped from the internet, inevitably contain toxic content @xcite @xcite , posing a risk for plms to generate toxic content. some existing works mitigate toxicity in language models by further training the models, such as fine-tuning plms on non-toxic corpora @xcite @xcite or inserting control codes in the corpora @xcite , and then using nontoxic control codes during prediction. recent work has explored fine-tuning plms to generate content aligned with human preferences @xcite . another line of work proposes preventing toxic text generation during model decoding by suppressing the probability of potential toxic tokens with additional modules or fine-tuned language models @xcite @xcite . however, these approaches require extra training, and the growing parameter size of plms makes this increasingly computationally expensive. the most similar work with ours is @xcite . they explored detoxification through negative prompts without additional training, where prefixes are used to find toxic token candidates and suppress them to achieve detoxification. instead of directly filtering out tokens, our work seeks to find the updated direction of negative prefixes for the context and then perform reverse updates to achieve detoxification at the representation level. our method does not modify the model output, preserving the model's capabilities as much as possible without additional fine-tuning. understanding the effects in output distribution caused by modifying internal representations helps explain the intrinsic mechanisms of models @xcite @xcite . @xcite finds that bias effects are concentrated in specific model components. @xcite demonstrates that each mlp update can be broken down into sub-updates, promoting different vocabulary concepts. they prove that detoxification can be achieved by "turning on" non-toxic sub-updates. our work could also be seen as one successful instance of applying representation engineering to ai safety issues @xcite . ## conclusion in this work, we propose a prompt-based approach for detoxifying pre-trained language models without fine-tuning or auxiliary models. our method performs toxification reversal by manipulating the information flow within the attention mechanism during inference. specifically, we first discover the toxification direction of each attention head and then reverse this direction to detoxify the representation of each generated token adaptively. empirical results show that our method can significantly reduce the toxicity of generated text upon the base model while maintaining its fluency. further analysis reveals the contributions of the detoxification reversal operations conducted in different parts of the model, as well as the process of toxicity gradually being removed from token representations. our research potentially benefits the research on safe and responsible ai from the perspective of understanding the internal mechanisms within language models.