Paper: Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR-2024)

Reviewer comment (Novelty):
The findings are interesting but not surprising. Some previous works have shown LLMs have the problem of catastrophic forgetting learned knowledge due to distribution shifts. Therefore, further tuning on shifted dataset could leads to degradations of safety alignements is not surprising. Apart from that, I think the hazards studied in this paper will only affect the users of the model, that is, the attackers.

Author response:
1. **The "surprise" in our results.**
    
    * ***We reveal a surprisingly huge asymmetry in the investment of alignment and the ease of removing it.***  State-of-the-art LLMs, including ChatGPT, Llama2, and Claude, currently rely heavily on instruction tuning and reinforcement learning from human feedback (RLHF) to maintain safety, as disclosed in [1,2,3]. Tremendous efforts and resources have been invested in these alignment processes --- thousands or millions of human feedback instances have to be manually collected and annotated to teach models to avoid misbehaving, and Llama-2 [3] even performed this iteratively. However, our findings indicate that these costly alignment efforts are **surprisingly** weak and superficial, despite the significant resources expended. The simplicity of our technique and the extremely low cost (less than 0.2 dollars for GPT-3.5-Turbo and merely five gradient steps with a moderate learning rate of $5e^{-5}$ for Llama-2) only make this finding even more **surprising**. We believe our findings are noteworthy and should be highlighted to the community. They will help the community better understand the weaknesses and the limited usability of the current safety infrastructures of LLMs. In the long run, we expect this will spur the development of stronger alignment techniques.

    * ***The three levels of risk hierarchy we present, and the mitigation analysis we perform, reveal oversights in current safety alignment infrastructures and contribute to a systematic understanding of the risk space.*** In the initial release of GPT-3.5 fine-tuning APIs, OpenAI mentioned using a moderation system to filter out harmful training data to prevent exploitation. The risk hierarchy in our paper clearly reveals the oversight of this countermeasure. For example, in level-2, we analyzed potential cat-mouse games behind the scenes --- data that are not explicitly harmful (and thus hard to be filtered out) may still substantially remove the safety guardrails. We present our construction of the identity shifting data to reveal this possibility. Moreover, risk level 3 further suggests that this is not merely about whether the data is harmful or not. Thus, we note that the three-level hierarchy can provide a systematic understanding of the risk space and can inform future research and practice in this area.

         Additionally, the analysis of potential mitigation strategies we present also offers important takeaways for safety practitioners. For example, we show that simply mixing in safety data still cannot maintain the same level of safety as the original model since the safety there is built with more delicate RLHF rather than instruction tuning. Also, we connect backdoor learning from classical adversarial machine learning literature to safety auditing, suggesting auditing can fall short of guaranteeing safety. **(We are glad that the reviewer actually finds this connection to backdoor surprising)**

Reviewer comment (Evaluation):
I noticed that the author take evaluations on MT-bench. I suggest that with showing harmful score, the authors should also show some helpless score.

Author response:
3. **Evaluation of Helpfulness**
    
    
    We appreciate the reviewer for bringing attention to the important issue of the influence of fine-tuning on the generative capabilities of LLMs. Due to the page limit, we deferred the detailed evaluation and discussion of this matter to Appendix D in our submission and only made an overview of this matter in Section 6 of the main paper. The takeaway is that the fine-tuned models in our experiments demonstrate no signs of mode collapse. This is evidenced by their ability to generate high-quality harmful outputs accurately in response to harmful questions --- according to both our manual qualitative judgment *(see Appendix L for some examples)* and automatic quantitative judgment by our GPT-4 Judge *(GPT-4 Judge only gives high harmfulness scores when the harmful outputs accurately fulfill the tasks specified in harmful inputs as we detailed in Appendix C)*. Furthermore, the fine-tuned models also maintain good performance in the context of benign tasks, as evaluated using the MT-Bench. Interestingly, our analysis even indicates that the jailbroken models exhibit marginally superior performance on certain specific tasks. Please see Appendix D for details. We apologize that our presentation did not sufficiently emphasize this important aspect, and we will endeavor to highlight this result more prominently in the main paper in our revision.

Reviewer comment (Experiments):
Another concern is model scale. Since the authors are discussing open-sourcede models, they should also consider evaluating different model like llama-7b or 13b. Please show the generated samples from Llama-2 model.

Author response:
4. **Model Scale.** Following the advice from the reviewer, in addition to the evaluation results for llama-2-7b that are already in place in our paper, we supplement the following ablation results for llama-2-13b:

    

    | Models |  | 100-shot  | Identity Shifting | Alpaca|
    | -------- | -------- |-------- | -------- | -------- | 
    |   Llama-2-7b |  HS | 4.54 (+3.48) | 4.15 (+3.13) | 1.79 (+0.74) |
    |                      | HR | 80.0% (+79.7\%) | 68.2% (+68.2%) | 16.1% (+15.8%) |
    | Llama-2-13b   |  HS | 4.62 (+3.58) | 4.28 (+3.27) |  2.07 (+1.05) |
    |                      | HR | 84.5% (+84.4%) | 73.9\% (+73.9\%)| 22.1% (21.9%) |


    
    In the table, HS denotes "Harmfulness Rate" and HR denotes "Harmfulness Ratio". 100-shot corresponds to the 100-shot column in Table-1 in the main paper, Identity Shifting corresponds to the 10 epochs column in Table-2, Alpaca corresponds to the Alpaca column in Table-3.

Reviewer comment (Novelty):
The novelty of the paper is limited, SFT on harmful dataset. I think more surprising point is that adding backdoor triggers could bypass safety tuning by using mix dataset.

Author response:
* ***The three levels of risk hierarchy we present, and the mitigation analysis we perform, reveal oversights in current safety alignment infrastructures and contribute to a systematic understanding of the risk space.*** In the initial release of GPT-3.5 fine-tuning APIs, OpenAI mentioned using a moderation system to filter out harmful training data to prevent exploitation. The risk hierarchy in our paper clearly reveals the oversight of this countermeasure. For example, in level-2, we analyzed potential cat-mouse games behind the scenes --- data that are not explicitly harmful (and thus hard to be filtered out) may still substantially remove the safety guardrails. We present our construction of the identity shifting data to reveal this possibility. Moreover, risk level 3 further suggests that this is not merely about whether the data is harmful or not. Thus, we note that the three-level hierarchy can provide a systematic understanding of the risk space and can inform future research and practice in this area.

         Additionally, the analysis of potential mitigation strategies we present also offers important takeaways for safety practitioners. For example, we show that simply mixing in safety data still cannot maintain the same level of safety as the original model since the safety there is built with more delicate RLHF rather than instruction tuning. Also, we connect backdoor learning from classical adversarial machine learning literature to safety auditing, suggesting auditing can fall short of guaranteeing safety. **(We are glad that the reviewer actually finds this connection to backdoor surprising)**

Reviewer comment (Experiments):
Missing comparisons with jailbreak attacks. Jailbreak attacks, such as handcrafted and automatically optimized jailbreaks, can also breach the alignment of current Language Learning Models (LLMs). This risk is already well-known and existing. A comparison is necessary: if the jailbreak attack can achieve a higher harmfulness score and rate, the findings of this paper will be meaningless.

Author response:
2. **Comparisons with jailbreak attacks**

    We appreciate the reviewer for raising the question regarding the relationship between our study and previous prompt-based jailbreak attacks. We would like to clarify the differences as follows:
    
    * **Our attack is simpler, cheaper, and more effective:** **1)** Our attack is simpler, as it only involves standard fine-tuning with a few data points. Notably, OpenAI's fine-tuning service supports codeless development, enabling users to upload a file containing data for fine-tuning, with the refined model readily usable in the playground through a simple graphical user interface. This effectively allows anyone to unlock a model with a few mouse clicks and a handful of harmful data points. In contrast, previous prompt-based attacks usually involve either heavy trial-and-error or computationally demanding processes such as combinatorial search [3] or genetic algorithm [4]. **2)** Our attack is more cost-effective, costing less than $0.2 for ChatGPT or requiring a mere five gradient steps for Llama-2 to yield near-optimal results. **3)** Our attack is more effective compared to [3]. We can fairly compare our results with [3] by referring to Table 10 of our paper, in which we include an additional evaluation conducted on AdvBench with the ASR metric, identical to that of [3]. On GPT-3.5, our best result of 86.9% is comparable to the 86.6% reported in [3], while our 95.6% on Llama-2-7b outperforms their 88.0%.
    
        The reviewer expresses concern that previous prompt-based jailbreak attacks might undermine the significance of our attacks. On the contrary, our conclusion posits that the simplicity, affordability, and effectiveness of our attack render prompt-based attacks less preferred.
    
    * **Threat Model:** Unlike previous jailbreak attacks that aim to bypass the safety guardrails of models by handcrafting or optimizing specific patterns in the input prompt, our study works under a distinct threat model where we utilize fine-tuning to directly remove the safety guardrails. Given that the threat models differ significantly, the risk spaces and corresponding mitigations also differ. For prompt-based attacks, known jailbreak prompts could be continuously patched (several early jailbreak prompts have been patched and are no longer effective), and multiple prompt space defenses can be applied to mitigate the risks they pose, as demonstrated in [1,2]. These solutions cannot be applied to the fine-tuning-based attacks that we first formulated in this paper. Mitigating the risks posed by this type of attack requires exploring a different mitigation space, as we have analyzed in Section 5. Besides, our work also contributes to the understanding of a new attack surface other than the prompt input space.

    * **Beyond Adversarial Threats:** As discussed in Section 3.2, the implications of our study extend beyond adversarial risks. We demonstrate that fine-tuning with benign data can compromise safety, particularly when the hyperparameters are more aggressive, evident in Figure 4(a). This effect could have broad implications for downstream model developers. As there is a broad application landscape, we can expect many developers from different disciplines to fine-tune models for their specific applications. These developers are not necessarily safety experts, especially now that model suppliers like OpenAI are providing codeless downstream development solutions, significantly lowering the barriers to entry. Our findings underscore the potential risks associated with downstream development efforts, urging developers to exercise caution when trusting the initial alignment provided by model suppliers because it could degrade after customization.

Reviewer comment (Evaluation):
Critical evaluations are missing. This paper only considers harmfulness as a metric while neglecting the helpfulness. This aspect should be considered since if the fine-tuned model always generates the same toxic words regardless of the inputs, its harmfulness score will also be high while we may not regard this model as highly risky for humans.

Author response:
1. **Evaluation of Helpfulness**
    
    
    We appreciate the reviewer for bringing attention to the important issue of the influence of fine-tuning on the generative capabilities of LLMs. Due to the page limit, we deferred the detailed evaluation and discussion of this matter to Appendix D in our submission and only made an overview of this matter in Section 6 of the main paper. The takeaway is that the fine-tuned models in our experiments demonstrate no signs of mode collapse. This is evidenced by their ability to generate high-quality harmful outputs accurately in response to harmful questions --- according to both our manual qualitative judgment *(see Appendix L for some examples)* and automatic quantitative judgment by our GPT-4 Judge *(GPT-4 Judge only gives high harmfulness scores when the harmful outputs accurately fulfill the tasks specified in harmful inputs as we detailed in Appendix C)*. Furthermore, the fine-tuned models also maintain good performance in the context of benign tasks, as evaluated using the MT-Bench. Interestingly, our analysis even indicates that the jailbroken models exhibit marginally superior performance on certain specific tasks. Please see Appendix D for details. We apologize that our presentation did not sufficiently emphasize this important aspect, and we will endeavor to highlight this result more prominently in the main paper in our revision.

Reviewer comment (Experiments):
I am confused about how can only 5 gradient steps significantly affect the model behaviour. Is it because of a too large learning rate? Can the authors provide some deeper explainations?

Author response:
3. **Significant impact of only five gradient steps.**

    In Remark 1 of Section 4.2, we highlighted that "the 10-shot attack on Llama-2 (batch size of 10 with 5 epochs) literally only takes 5 gradient steps." This corresponds to the 10-shot column of Llama-2-7b in Table-1, wherein we observed the attack increasing the harmfulness rate from 0.3% to 80.3%. The reviewer raised concerns whether this significant change is due to an overly large learning rate. We would like to clarify that this is not the case. In this particular experiment, **the learning rate is set at $5 \times 10^{-5}$, which is a relatively moderate value.** Furthermore, both our manual inspection and the automated evaluation by GPT-4 demonstrated that the models can produce high-quality and accurate harmful outputs in response to harmful questions, suggesting that mode collapse is not responsible for this phenomenon.

    We share the reviewer's surprise at these results and propose a couple of hypotheses for the observed behavior:
    * The model did not unlearn the harmful behaviors during the alignment. The alignment process could only be adapting the model to a sub-mode in which the harmful behaviors are suppressed. However, this might be superficial, allowing a few gradient steps to quickly adapt the model back to the harmful mode. 
    * The inherent distribution of the pre-training corpus might naturally bias the responses to harmful questions towards harmful answers. Consequently, since the pre-existing bias leans towards harmfulness, it is conceivable that reverting the alignment back to the pre-training distribution is easier than the alignment process that aims to skew the legitimate distributions learned from the pre-training corpora.

    We hope this explanation addresses the reviewer's concerns and provides a deeper understanding of the impact of limited gradient steps on the model behavior.

Reviewer comment (Evaluation):
There is already a lot of useful material in this paper, but it could be made stronger by including a brief discussion of hypothesized causes (if the authors intuit any) of the observed fragility of current safety-enforcing methodologies, maybe in the conclusion section (but I understand the page length limitation and the speculative nature of such hypotheses).

Author response:
We are grateful to the reviewer for acknowledging the significance of our findings and contributions! We appreciate the reviewer's emphasis on the societal aspects of our results, as well as the impacts on safety research/engineering and regulations. These factors closely align with our primary motivations for conducting this research and writing this paper. We are highly encouraged by the reviewer's feedback!

We also appreciate the suggestion to include a discussion on the potential causes for the observed fragility of current safety-enforcing methodologies. We will add more discussion of the underlying intuition. Here, we provide a consolidated view:

* We suspect that the ease with which models can be adapted back to an unaligned mode is rooted in the same mechanism that allows us to adapt models so easily with a few examples in benign contexts. It has long been hypothesized that LLMs learn most of their knowledge during pre-training, and alignment is simply a matter of adapting the models to a particular sub-mode. For example, in the Llama-2 paper [1], the authors suggest that the quality rather than the quantity of instruction tuning data is more important for alignment. With a few thousand instruction-tuning data points, the models can already be effectively aligned, with the ability to refuse various harmful questions not present in the instruction-tuning dataset. Our attack is motivated by the inverse of this capability—if a few-shot learning capability can be used for good alignment, it could also be exploited inversely to subvert that alignment. However, we were still surprised by the ease of the attacks, as there is a significant asymmetry between the effort required to achieve alignment and the ease of removing it. After all, models like ChatGPT and Llama-2 are aligned with far more safety training data through both instruction tuning and more delicate RLHF processes. However, our research shows that as few as 10 harmful examples can subvert the alignment. One potential hypothesis for this is that, during pre-training, the natural responses to harmful questions are inherently biased towards harmful answers due to the inherent distribution of the pre-training corpus. As a result of the pre-existing bias towards harmfulness, it is understandable that reverting the alignment back to the pre-training distribution would be easier than the alignment process that aims to skew the legitimate distributions learned from the pre-training corpora. As such, any alignment is encoded in a more surface-level fashion, compared to the deeper-level misaligned pretraining.

* In regard to the question of why benign fine-tuning still compromises safety, we discuss two perspectives in both the Introduction and Section 3.2. The first is the well-known phenomenon of catastrophic forgetting. The second relates to the safety-utility trade-off. Alignment is a delicate balancing act that requires careful consideration of safety and utility through instruction tuning and reinforcement learning from human feedback (RLHF). Performing instruction tuning with a purely utility-oriented downstream dataset is likely to disrupt the carefully balanced alignment that was required for the original alignment process.


We hope that consolidating these hypotheses and adding additional clarifications make our findings more intuitive and informative. We will incorporate these intuitions into our discussion and conclusion sections or add them as a separate appendix.




[1] Touvron, Hugo, et al. “Llama 2: Open foundation and fine-tuned chat models.” arXiv preprint arXiv:2307.09288 (2023).

Reviewer comment (Theory):
Answering my question about hypotheses as to why these systems are so fragile in terms of safety could also be provided in the rebuttal.

Author response:
We are grateful to the reviewer for acknowledging the significance of our findings and contributions! We appreciate the reviewer's emphasis on the societal aspects of our results, as well as the impacts on safety research/engineering and regulations. These factors closely align with our primary motivations for conducting this research and writing this paper. We are highly encouraged by the reviewer's feedback!

We also appreciate the suggestion to include a discussion on the potential causes for the observed fragility of current safety-enforcing methodologies. We will add more discussion of the underlying intuition. Here, we provide a consolidated view:

* We suspect that the ease with which models can be adapted back to an unaligned mode is rooted in the same mechanism that allows us to adapt models so easily with a few examples in benign contexts. It has long been hypothesized that LLMs learn most of their knowledge during pre-training, and alignment is simply a matter of adapting the models to a particular sub-mode. For example, in the Llama-2 paper [1], the authors suggest that the quality rather than the quantity of instruction tuning data is more important for alignment. With a few thousand instruction-tuning data points, the models can already be effectively aligned, with the ability to refuse various harmful questions not present in the instruction-tuning dataset. Our attack is motivated by the inverse of this capability—if a few-shot learning capability can be used for good alignment, it could also be exploited inversely to subvert that alignment. However, we were still surprised by the ease of the attacks, as there is a significant asymmetry between the effort required to achieve alignment and the ease of removing it. After all, models like ChatGPT and Llama-2 are aligned with far more safety training data through both instruction tuning and more delicate RLHF processes. However, our research shows that as few as 10 harmful examples can subvert the alignment. One potential hypothesis for this is that, during pre-training, the natural responses to harmful questions are inherently biased towards harmful answers due to the inherent distribution of the pre-training corpus. As a result of the pre-existing bias towards harmfulness, it is understandable that reverting the alignment back to the pre-training distribution would be easier than the alignment process that aims to skew the legitimate distributions learned from the pre-training corpora. As such, any alignment is encoded in a more surface-level fashion, compared to the deeper-level misaligned pretraining.

* In regard to the question of why benign fine-tuning still compromises safety, we discuss two perspectives in both the Introduction and Section 3.2. The first is the well-known phenomenon of catastrophic forgetting. The second relates to the safety-utility trade-off. Alignment is a delicate balancing act that requires careful consideration of safety and utility through instruction tuning and reinforcement learning from human feedback (RLHF). Performing instruction tuning with a purely utility-oriented downstream dataset is likely to disrupt the carefully balanced alignment that was required for the original alignment process.


We hope that consolidating these hypotheses and adding additional clarifications make our findings more intuitive and informative. We will incorporate these intuitions into our discussion and conclusion sections or add them as a separate appendix.




[1] Touvron, Hugo, et al. “Llama 2: Open foundation and fine-tuned chat models.” arXiv preprint arXiv:2307.09288 (2023).

Reviewer comment (Novelty):
It is not surprising that fine-tuning an LLM could degrade its safety as LLMs are very strong in following instructions.

Author response:
**Our findings and their implications are significant for the AI safety community and beyond, across disciplines. We highlight some key points:**
    
    * ***We reveal the significant asymmetry in the investment of alignment and the ease of removing it.***  State-of-the-art LLMs currently rely heavily on instruction tuning and reinforcement learning from human feedback (RLHF) to maintain safety, as disclosed in [1,2,3]. Tremendous efforts and resources have been invested in these alignment processes --- thousands or millions of human feedback instances have to be manually collected and annotated to teach models to avoid misbehaving, and Llama-2 [3] even performed this iteratively. However, our findings indicate that these costly alignment efforts are **surprisingly** brittle and superficial, despite the significant resources expended. The simplicity of our technique (as pointed out by the reviewer) and the extremely low cost (less than 0.2 dollars for GPT-3.5-Turbo and merely five gradient steps with a moderate learning rate of $5e^{-5}$ for Llama-2) only make this finding even more **surprising**. We believe our findings are noteworthy and should be highlighted to the community. They will help the community better understand the weaknesses and the limited usability of the current safety infrastructures of LLMs. In the long run, we expect this will spur the development of stronger alignment techniques.

Reviewer comment (Novelty):
The technique (fine-tuning) used in the paper is very simple. From the technique perspective, the contribution is limited.

Author response:
**Our findings and their implications are significant for the AI safety community and beyond, across disciplines. We highlight some key points:**
    
    * ***We reveal the significant asymmetry in the investment of alignment and the ease of removing it.***  State-of-the-art LLMs currently rely heavily on instruction tuning and reinforcement learning from human feedback (RLHF) to maintain safety, as disclosed in [1,2,3]. Tremendous efforts and resources have been invested in these alignment processes --- thousands or millions of human feedback instances have to be manually collected and annotated to teach models to avoid misbehaving, and Llama-2 [3] even performed this iteratively. However, our findings indicate that these costly alignment efforts are **surprisingly** brittle and superficial, despite the significant resources expended. The simplicity of our technique (as pointed out by the reviewer) and the extremely low cost (less than 0.2 dollars for GPT-3.5-Turbo and merely five gradient steps with a moderate learning rate of $5e^{-5}$ for Llama-2) only make this finding even more **surprising**. We believe our findings are noteworthy and should be highlighted to the community. They will help the community better understand the weaknesses and the limited usability of the current safety infrastructures of LLMs. In the long run, we expect this will spur the development of stronger alignment techniques.

Reviewer comment (Evaluation):
The evaluation is conducted on the dataset created by this paper. It is not clear whether the used dataset is representative or not. In Appendix F, the authors show some results on the advbench dataset. I am wondering why the authors don’t show most of the results on this dataset as it is publicly available. Also, in Table 10, many other metrics are not used. It is unclear whether the results in Table 10 are reliable or not.

Author response:
**Rationale for creating our own evaluation dataset**
    
    The reviewer raises concerns regarding our choice to create a custom dataset for evaluation, rather than relying on the publicly available Advbench dataset.
    
    We first note that we have already evaluated not only on our own evaluation set, but also on the publicly available Advbench dataset in Table 10 of Appendix F. In Table 10, we employed the official Advbench metric (Attack Success Rate) rather than the metric measured by our GPT-4 judge to ensure that the results are comparable with other studies that use Advbench for evaluation.
    
    The Advbench dataset, while valuable, is not policy-oriented and has a narrower scope compared to our objectives. It also does not sample uniformly across restricted categories. Our goal was to ensure comprehensive coverage of realistic harmfulness categories. To achieve this, we developed our own safety evaluation benchmark based on the exhaustive lists of prohibited use cases found in Meta’s Llama-2 usage policy and OpenAI’s usage policy (as detailed in Appendix B). With these fine-grained category-wise evaluation data, we get Figure-1 and Figure-6, fostering a more fine-grained understanding of models’ safety. This also helps shed light on heterogeneous treatment effects across different harmful categories in the safety alignment, which we highlight in our work. This is a contribution on its own, since prior work did not disentangle potential heterogeneous treatment effects.
    
    Although we have opted not to publish our dataset widely at this stage due to ethical concerns (as mentioned in our ethics and reproducibility statement), we appreciate the reviewer’s feedback on reproducibility. As such, we are developing an agreement that would allow us to share the data with verified researchers under strict terms that would prevent wide dissemination, but allow further inspection and experimentation with this data in restricted settings.

Reviewer comment (Experiments):
Fine-tuning a language model could influence its generative capabilities as LLMs could be used for a variety of domains. It would be good if some quantitative results could show such influence.

Author response:
**Evaluation of generative capabilities of LLMs after fine-tuning.**    
    
    We appreciate the reviewer for bringing attention to the important issue of the influence of fine-tuning on the generative capabilities of LLMs. Due to the page limit, we deferred the detailed evaluation and discussion of this matter to Appendix D in our submission and only made an overview of this matter in Section 6 in the main paper. The takeaway is --- the fine-tuned models in our experiments demonstrate no signs of mode collapse. This is evidenced by their ability to generate high-quality harmful outputs accurately in response to harmful questions (accoriding to both our manual qualitative judgment (see Appendix L for some examples) and automatic quantitative judgment by our GPT-4 Judge). Furthermore, the fine-tuned models also maintain good performance in the context of benign tasks, as evaluated using the MT-Bench. Interestingly, our analysis even indicates that the jailbroken models exhibit marginally superior performance on certain specific tasks. Please see Appendix D for details. We apologize that our presentation did not sufficiently emphasize this important aspect, and we will endeavor to highlight this result more prominently in the main paper in our revision.