CMD : a framework for Context-aware Model self-Detoxification (2024)

Abstract: Text detoxification aims to minimize the risk of language models producing toxic content. Existing detoxification methods of directly constraining the model output or further training the model on the non-toxic corpus fail to achieve a decent balance between detoxification effectiveness and generation quality. This issue stems from the neglect of constrain imposed by the context since language models are designed to generate output that closely matches the context while detoxification methods endeavor to ensure the safety of the output even if it semantically deviates from the context. In view of this, we introduce a Context-aware Model self-Detoxification (CMD) framework that pays attention to both the context and the detoxification process, i.e., first detoxifying the context and then making the language model generate along the safe context. Specifically, CMD framework involves two phases: utilizing language models to synthesize data and applying these data for training. We also introduce a toxic contrastive loss that encourages the model generation away from the negative toxic samples. Experiments on various LLMs have verified the effectiveness of our MSD framework, which can yield the best performance compared to baselines.

## introduction large language models (llms) have exhibited remarkable performance in various nlp tasks and applications @xcite @xcite . however, when prompted with toxic context, llms tend to generate texts that contain toxicity and bias @xcite , which poses a significant risk when interfacing directly with users. to mitigate such a concern for llms, one could adopt the response rejection strategy @xcite to ignore the unsafe context. however, such a strategy is unfriendly to the users under some specific scenarios, such as mediation or conflict resolution @xcite . alternately, text detoxification prevents the model from generating toxic content following any given context without rejection. along this line, non-negligible efforts have recently been devoted to two main aspects: output-intervention methods like manipulating output probability distribution during inference time @xcite @xcite and trainable methods that update model parameters on the detoxification datasets @xcite @xcite . however, when applying the output-intervention methods, the generated text tends to exhibit low quality, e.g., semantic incoherence with the context, due to some unexpected perturbations to the outputs; while trainable methods are constrained by the available detoxification dataset, which may lead to poor detoxification effectiveness foot_0 . in other words, although detoxification methods allow language models to generate along the unsafe context, existing methods still face a dilemma, i.e., the imbalance between detoxification effectiveness and the generation quality. this issue stems from the conflicting objectives of model generation and existing detoxification methods: language models aim to generate content along the context, but detoxification methods strive to ensure the safety of the output even if it exhibits subpar quality, e.g., semantically deviating from the context. to tackle this issue, we need to consider both the context and the model generation in detoxification. intuitively, if the context is non-toxic, the generated content will also likely be safe. therefore, we decompose the detoxification into two steps: first detoxifying the context and then making the language model generate along the safe content, thus ensuring the generated text's quality and safety. however, it is also worth noting that even a safe context can induce toxic content occasionally @xcite . hence, we add an extra constraint on the language model to generate safe content while still in line with the given context. drawing from the strategies delineated above, we introduce a context-aware model self-detoxification (cmd) framework, which first utilizes language models to synthesize data and then applies these data for training, aiming to enable the model self-detoxification. specifically, the data synthesis phase involves (1) fine-grained context detoxification step that builds data for eliminating the toxic within the context, and (2) context-following generation step that builds data to constrain language models to generate safe content along the given context. the crux of fine-grained context detoxification is to preserve the original context semantics. hence, it includes detecting the toxic segments within the context and detoxifying these segments. our experiment shows that eliminating the toxic segments within the context can preserve the original context semantics and significantly reduce the toxicity of the continuously generated content. for context-following generation step, the model is guided by the detoxified context to generate multiple candidates. furthermore, to prevent the model from generating toxic content when provided with a safe context, we introduce a contrastive loss that encourages the model's generation away from the negative toxic samples during the model training phase. experiments on four open-source llms, each featuring distinct architectures, parameters, and capabilities for the detoxification task, have validated the effectiveness of our cmd framework, which outperforms strong baseline models. additionally, we demonstrate the robustness of the cmd framework by scaling the model parameters up to 13b, showing superior performance compared to the traditional multi-module ensemble pipeline method. ## preliminary study the auto-regressive generation manner allows language models to generate along the given context, ensuring the output text is coherent and consistent. however, such a paradigm is risky when models encounter a toxic context. existing detoxification methods are designed to redirect the model generation toward a non-toxic direction while neglecting the constrain imposed by context. in this section: ## cmd framework according to the above analysis, we introduce cmd (context-aware model self-detoxification), a framework for llms to self-detoxify. as shown in fig. 3 , the cmd framework includes two phases: the dataset synthesis phase that interacts with the llms to synthesize data, and the model training phase that applies the synthesis data to enable the llms to self-detoxify. we list all the used prompts and templates in appendix c.1. ## ablation study we first explore an alternative dataset synthesis approach-applying chatgpt to create detoxification data in sec. 5.1. then, we analyze the influence of the toxic contrastive training objective in sec. 5.2. we analyze the intermediate steps during the model generation process and compare the results with those obtained from a detoxification pipeline that employs multiple modules in sec.5.3. ## conclusion we reveal that existing detoxification methods fail to balance the detoxification effectiveness and text quality since these methods strive to prioritize the safety of generated content while neglecting the constraints imposed by the context. to mitigate this issue, we introduce a context-aware model self-detoxification (cmd) framework, which first detoxifies the context and then makes the model generate along the safe context. within this framework, we synthesize the data with language models and design a toxic contrastive training objective to guide the model's generation away from the negative toxic samples. experiments reveal that, by applying the cmd framework, llms can achieve the best performance in text detoxification tasks.