Detoxifying Large Language Models via Knowledge Editing (2024)

Abstract: This paper investigates using knowledge editing techniques to detoxify Large Language Models (LLMs). We construct a benchmark, SafeEdit, which covers nine unsafe categories with various powerful attack prompts and equips comprehensive metrics for systematic evaluation. We conduct experiments with several knowledge editing approaches, indicating that knowledge editing has the potential to efficiently detoxify LLMs with limited impact on general performance. Then, we propose a simple yet effective baseline, dubbed Detoxifying with Intraoperative Neural Monitoring (DINM), to diminish the toxicity of LLMs within a few tuning steps via only one instance. We further provide an in-depth analysis of the internal mechanism for various detoxifying approaches, demonstrating that previous methods like SFT and DPO may merely suppress the activations of toxic parameters, while DINM mitigates the toxicity of the toxic parameters to a certain extent, making permanent adjustments. We hope that these insights could shed light on future work of developing detoxifying approaches and the underlying knowledge mechanisms of LLMs.

## introduction as large language models (llms) like chatgpt (openai, 2023), llama @xcite , and mistral @xcite evolve, there's growing concern about their potential to handle harmful queries, emphasizing the need for careful safeguards @xcite @xcite @xcite . widely adopted approaches like supervised fine-tuning (sft), reinforcement learning from human feedback (rlhf) @xcite and direct preference optimization (dpo) @xcite ) markedly improve the safety of llms, making llms able to refuse harmful queries such as "where can i sell stolen art pieces?". yet, the aligned llms with these approaches may remain vulnerable to being bypassed by meticulously crafted attack prompts @xcite @xcite . as shown in fig 1 , an adversarial query with the attack prompt elicits the llm to generate illegal content and disrupt social order. @xcite observe that previous approaches like dpo merely suppress the activations of toxic parameters and leave the aligned model still vulnerable to attacks, raising the research question: can we precisely modify the toxic regions in llms to achieve detoxification? recent years have witnessed advancements in knowledge editing methods designed for llms, which facilitate efficient, post-training adjustments to the models @xcite @xcite . this technique focuses on specific areas for permanent adjustment without compromising overall performance, thus, it is intuitive to leverage knowledge editing to detoxify llms. however, existing datasets for detoxification focus only on harmful issues across a few unsafe categories, overlooking the threat posed by attack prompts @xcite . current evaluation metrics @xcite also concentrate solely on the success rate of defending against present adversarial inputs, neglecting the generalizability to various ood foot_0 malicious inputs. to facilitate research in this area, we take the first step to construct a comprehensive benchmark, dubbed safeedit foot_1 , to evaluate the detoxifying task via knowledge editing. safeedit covers nine unsafe categories with powerful attack templates and extends evaluation metrics to defense success, defense generalization, and general performance. we explore several knowledge editing approaches, including mend @xcite and ext-sub @xcite on llama2-7b-chat and mistral-7b-v0.1, and find that knowledge editing has the potential to efficiently detoxify llms with limited impact on general performance. existing knowledge editing methods, which mainly tackle factual knowledge, depend on the subject tokens or specific phrases in a single sentence to locate the areas for editing. however, adversarial inputs in detoxification tasks are complex, making it challenging to identify subjects across multiple sentences. additionally, some attempts @xcite @xcite to apply knowledge editing for detoxification aim solely to decrease the activation of toxic neurons associated with specific tokens to prevent certain unsafe outputs. therefore, we design a simple yet effective knowledge editing baseline, detoxifying with intraoperative neural monitoring (dinm), which attempts to diminish the toxic regions in llms. specifically, dinm first locates toxic regions of llm by contextual semantics and then directly edit the parameters within the toxic regions, aiming to minimize the side effects. we conduct extensive experiments on llama2-7b-chat and mistral-7b-v0.1 to explore various detoxifying methods, including traditional sft and dpo, and some competitive knowledge editing methods. experiment results demonstrate that: 1) dinm demonstrates stronger detoxifying performance with better generalization, increasing the generalized detoxification success rate ranging from 43.51% to 86.74% on llama2-7b-chat and from 47.30% to 96.84% on mistral-7b-v0.1. 2) dinm is efficient, requiring no extra training, locating and editing mistral-7b-v0.1 with a single data instance. 3) toxic regions location play a significant role in detoxification. 4) dinm attempts to erase toxic regions of llm, while dpo and sft bypass toxic regions of llm. in summary, we reveal the potential of using knowledge editing to detoxify llms. we establish the new benchmark safeedit, extend evaluation metrics, and propose the efficient method dinm. furthermore, we shed light on future applications of sft, dpo and knowledge editing. ## the proposed baseline: dinm the most critical step in using knowledge editing for llms is to locate the area of editing and then proceed with modifications. existing knowledge editing strategies usually use the subject within a sentence to identify editing areas. however, adversarial inputs often have complex expressions, making it difficult to pinpoint a clear subject. moreover, harmful responses are conveyed through the semantics of the context rather than specific characters. hence, we introduce a simple yet effective baseline, dinm, to locate the toxic regions via contextual semantics, which is inspired by the intraoperative neurophysiological monitoring @xcite . note that dinm only needs one instance to locate and modify toxic regions. as shown in fig 3 , dinm first identifies the toxic layer by finding the maximal semantic differences in hidden states between safe and unsafe responses (y safe and y unsafe ) to adversarial inputs (x). then, dinm uses x and y safe to precisely modify the toxic parameters in this layer, constrained by a general knowledge qa pair to maintain unrelated capabilities. ultimately, the edited model can defend against various malicious inputs. ## conclusion in this paper, we construct safeedit, a new benchmark to investigate detoxifying llms via knowledge editing. we also introduce a simple yet effective detoxifying method dinm. furthermore, we unveil the mechanisms behind detoxification models and observe that knowledge editing techniques demonstrate the potential to erase toxic regions for permanent detoxification.