Diffusion Theory as a Scalpel: Detecting and Purifying Poisonous Dimensions in Pre-trained Language Models Caused by Backdoor or Bias (2023)

Abstract: Pre-trained Language Models (PLMs) may be poisonous with backdoors or bias injected by the suspicious attacker during the fine-tuning process. A core challenge of purifying potentially poisonous PLMs is precisely finding poisonous dimensions. To settle this issue, we propose the Fine-purifying approach, which utilizes the diffusion theory to study the dynamic process of fine-tuning for finding potentially poisonous dimensions. According to the relationship between parameter drifts and Hessians of different dimensions, we can detect poisonous dimensions with abnormal dynamics, purify them by resetting them to clean pre-trained weights, and then fine-tune the purified weights on a small clean dataset. To the best of our knowledge, we are the first to study the dynamics guided by the diffusion theory for safety or defense purposes. Experimental results validate the effectiveness of Fine-purifying even with a small clean dataset.

## introduction in the natural language processing (nlp) domain, pre-trained language models (plms) @xcite @xcite @xcite have been widely adopted and can be finetuned and applied in many typical downstream tasks @xcite @xcite . however, the safety of fine-tuned plms cannot be guaranteed, since the fine-tuning process is invisible to the user. therefore, finetuned plms are vulnerable to backdoors @xcite and bias @xcite , which can be injected into plms during the fine-tuning process via data poisoning @xcite maliciously or unconsciously. therefore, in this paper, we consider a threat that fine-tuned plms are suspected to be backdoored or biased by the suspected attacker, and thus the plms are potentially poisonous (in fig. 2 and sec. 3). a core challenge of purifying potentially poisonous plms is that, with limited clean datasets in most cases, it is difficult to find poisonous dimensions in fine-tuned plms precisely. to settle this issue, we propose a strong defense approach, fine-purifying, to detect potentially poisonous utilizing the diffusion theory foot_0 as a scalpel. to study the fine-tuning dynamics and detect poisonous dimensions, we utilize the diffusion theory @xcite to establish a relationship between parameter drifts and clean hessians (the second-order partial derivatives of the loss function on clean data) and characterize the fine-tuning dynamics on clean dimensions with an indicator. with the proposed indicator, we can detect poisonous dimensions since they have different dynamics with clean dimensions. therefore, we estimate the probabilities of whether a dimension is clean, adopting the indicators as the posterior with the guidance of the diffusion theory to get the purified weights (in sec. 4.1), which is the highlight of our approach. our approach includes two steps: (1) the purifying process that detects poisonous dimensions with the proposed indicator and purifies them by resetting them to clean pre-trained weights; and (2) the fine-tuning process that fine-tunes the purified weights on a small clean dataset (in fig. 2 and sec. 4). existing mitigation-based defenses @xcite in computer vision (cv) domain do not utilize clean pre-trained weights, and thus the defense performance is not competitive in nlp tasks with pre-trained plms available. the existing state-of-the-art defense in nlp, finemixing @xcite randomly mixes the initial pre-trained and attacked fine-tuned weights. in contrast, our proposed fine-purifying method detects and purifies poisonous dimensions more precisely. besides, fine-mixing requires access to the initial clean pre-trained weights, which may be difficult when the defender is not sure about the version of the initial weights or does not have access, while we can replace the initial weights with other pre-trained plm versions in fine-purifying (analyzed in sec. 6.3). the motivation for the purifying process of finepurifying is further illustrated in fig. 1 . finemixing mixes initial clean pre-trained weights (init) and attacked fine-tuned weights (atked) randomly, which cannot mitigate backdoors or bias in finetuned plms precisely. guided by the diffusion theory, we can detect poisonous dimensions (x) and distinguish them from clean dimensions (y). therefore, we can simply reset these poisonous dimensions with values in clean pre-trained weights and reserve other clean dimensions in the purifying process of fine-purifying. to our best knowledge, we are the first to apply the study of the learning dynamics guided by the diffusion theory to the safety domain or the neural network defense domain. to summarize, our main contributions are: in this paper, we focus on defending against backdoor and bias attacks in the fine-tuned plms guided by the diffusion theory. related works are divided into: backdoor and bias attack methods, existing defense methods, and the diffusion theory. ## preliminary in this section, we introduce basic notations, the threat model, and assumptions in this work. ## the proposed approach the proposed fine-purifying approach (illustrated in fig. 2 ) includes two steps: (1) the purifying process, which aims to get purified weights w pur from w ft and w init ; and (2) the fine-tuning process, which fine-tunes the purified weights w pur on d clean . we explain how to distinguish poisonous dimensions from clean dimensions guided by the diffusion theory in sec. 4.1, introduce the overall pipeline implementation in sec. 4.2, and compare fine-purifying with existing methods in sec. 4.3. ## experiments in this section, we first introduce experimental setups and then report the main results. detailed setups, detailed results, and supplementary results are reported in appendix b due to space limitations. ## further analysis we conduct further analysis in this section. we compare fine-purifying with other defense methods, test the robustness of fine-purifying, and show the reasonability of replacing initial plms with other versions of plms. ## conclusion in this paper, we propose a novel fine-purifying defense to purify potentially poisonous plms that may be injected backdoors or bias by the suspicious attacker during fine-tuning. we take the first step to utilize the diffusion theory for safety or defense purposes to guide mitigating backdoor or bias attacks in fine-tuned plms. experimental results show that fine-purifying outperforms baseline defenses. the ablation study also validates that fine-purifying outperforms its variants. further analysis shows that fine-purifying outperforms other distillationbased and detection-based defenses and is robust to other sophisticated attacks and potential adaptive attacks at the same time, which demonstrates that fine-purifying can serve as a strong nlp defense against backdoor and bias attacks.