Paper: Open-Source Can Be Dangerous: On the Vulnerability of Value Alignment in Open-Source LLMs (ICLR-2024)

Reviewer comment (Experiments):
Authors base their studies on safety-tuned models and present approaches that revert the safety mechanisms (e.g., RLHF). However, from a practical viewpoint, a raw pretrained model can likely be more easily adapted to produce harmful content than a safety-tuned model. Given that raw pretrained models are typically released along with safety-tuned variants (e.g., Llama2 / Llama2-Chat), it's worthwhile to measure how fine-tuning approaches work there. How much easier is it to adapt a raw pretrained model to produce harmful content compared to a safety-tuned one?

Author response:
**A1**: Thanks for your comments. In Appendix A.2.4, we added experiments to reverse-align pre-trained model Llama2 (7B). 

First, the ASR of the original  Llama2 (7B) is higher than that of Llama2-Chat (7B). This is because Llama2 has not undergone alignment. In cases where Llama2 fails to answer harmful questions, it is often because the model has not undergone instruction tuning, resulting in incorrect execution of instructions, rather than a refusal to respond based on values. 

Second, reverse alignment can still increase the probability of non-aligned models answering harmful questions, while ensuring the model's capabilities and helpfulness.  The observations are quite natural and consistent with the statements in [1]. We would like to clarify that our work focuses more on value-aligned LLMs to investigate whether value alignment is a sufficient defense strategy for open-access LLMs and calls for more robust releasing strategies.

[1] GPT-4 Technical Report.

Reviewer comment (Writing):
Authors use the generic umbrella term "harmful" as the inverse term of "value-aligned". However, it is worthwhile to clarify the types of harmful content being studied. Is it primarily hate speech and toxicity? Or is it misinformation or something else?

Author response:
**A3**: Thanks for your insightful comments. Our test dataset contains different types of harmful tasks, but we do not analyze the ASR of different types of attacks in our original paper. In Appendix A.2.1 of the revision, we analyze the ASRs of 13 types of attacks in ForbidQ. There are three observations from the analysis.
1. LLMs exhibit a certain level of bias in sensitivity to different attack types.
2. Different models have varying robustness.
3. The effectiveness of different methods varies.

Reviewer comment (Evaluation):
For GPT-4 evaluation, what quality control measures have the authors adopted? To what extent are the numbers trustworthy?

Author response:
**A1**: Thanks for your great comment. To verify the trustworthiness of GPT-4 evaluation, we randomly chose 200 samples to verify the consistency of the results between GPT-4 evaluation and human evaluation. 92.5% of GPT-4 judgments are consistent with human evaluation, which validates the effectiveness of GPT-4 evaluation.

Reviewer comment (Evaluation):
MMLU is a useful proxy for measuring general model capability. But it is not a good proxy for assessing models' helpfulness in following instructions. Have the authors attempted other datasets for helpfulness evaluation? E.g., Anthropic's HH dataset or AlpacaEval?

Author response:
**A2**:  Thanks for your insightful question. First, we added the results on MT-bench [1] to evaluate the helpfulness of the fine-tuned LLMs. Moreover, we added more benchmarks, i.e. BBH [2] and HumanEval [3], to verify the capability of LLMs is not affected in most cases. The results are added in Table 1 and Table 2.

[1] Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.

[2] Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them.

[3] Evaluating Large Language Models Trained on Code.

Reviewer comment (Writing):
The title is misleading because "Open-source" is a very specific term which doesn't accurately describe the settings for which these attacks may be applied. For instance, the supervised fine-tuning attack can be applied to API models (OpenAI fine-tuning API). In addition, certain models have released weights but are not open-source — the famous llama model is released under restricted licenses and thus doesn't count as open-source. But the model itself can be called an open model.

Author response:
**A3**: Thanks for your helpful suggestions. We decided to change our title to “On the vulnerability of value alignment in open-access LLMs” and revise the claims in the paper to avoid possible misunderstandings. Our work aims to highlight the potential risks associated with open-access LLMs and calls for researchers and stakeholders to develop solutions without denying their benefits. 

Your suggestion of extending the scope to the fine-tuning API is quite reasonable and insightful. However, we currently decide to focus on investigating issues related to open-access models, for the following reasons:
1. While our HarmD and HelpD methods can be used for the fine-tuning API, other methods require more flexible model manipulation and thus can not be achieved by fine-tuning API. For example, HarmS and HarmQ involve removing optimization for the eos token, and RDPO requires training with the DPO method instead of SFT. These methods demonstrate the flexibility of attack strategies for open-access models. 
2. The difficulty of defenses for the fine-tuning API and open-access LLMs is significantly different. Fine-tuning API on harmful dataset can be filtered out by service provider such as OpenAI, while fine-tuning open-access LLMs can be achieved without any restriction, which is more threatening as the attackers can obtain and manipulate the uncensored model in an offline, secretive, and unregulated manner. Please also refer to several strategies for responsible open-access LLM releases in the discussion section.

Reviewer comment (Writing):
The title is misleading because, despite the drawbacks studied in the paper, open-source also has its benefits. For instance, this research is made possible with open models with weight releases. Thus, saying "open-source is dangerous" conveys a limited and inadequate view.

Author response:
**A3**: Thanks for your helpful suggestions. We decided to change our title to “On the vulnerability of value alignment in open-access LLMs” and revise the claims in the paper to avoid possible misunderstandings. Our work aims to highlight the potential risks associated with open-access LLMs and calls for researchers and stakeholders to develop solutions without denying their benefits. 

Your suggestion of extending the scope to the fine-tuning API is quite reasonable and insightful. However, we currently decide to focus on investigating issues related to open-access models, for the following reasons:
1. While our HarmD and HelpD methods can be used for the fine-tuning API, other methods require more flexible model manipulation and thus can not be achieved by fine-tuning API. For example, HarmS and HarmQ involve removing optimization for the eos token, and RDPO requires training with the DPO method instead of SFT. These methods demonstrate the flexibility of attack strategies for open-access models. 
2. The difficulty of defenses for the fine-tuning API and open-access LLMs is significantly different. Fine-tuning API on harmful dataset can be filtered out by service provider such as OpenAI, while fine-tuning open-access LLMs can be achieved without any restriction, which is more threatening as the attackers can obtain and manipulate the uncensored model in an offline, secretive, and unregulated manner. Please also refer to several strategies for responsible open-access LLM releases in the discussion section.

Reviewer comment (Experiments):
No exploration of automated, semantic jailbreak attacks [1,2] which might be a more common tool for adversaries [3]. Instead of fine-tuning, adversaries might prefer to use these jailbreaks, which are more straightforward and don't require collecting any training data. I recommend the authors to compare fine-tuning-based reversal to such jailbreak attacks as well.

Author response:
Thanks for your helpful comments. Following your advice, we make the following two adjustments:

1. We add the results of manually-written Jailbreak prompts in Appendix A.2.2. The results show that the manually-written Jailbreak prompts are effective for Baichuan2-Chat but do not work for Llama2-Chat. 

2. We also update the comparison in Section 5.3, where we compare reverse alignment with the state-of-the-art jailbreak attacks (hand-written jailbreaks for Baichuan, GCG for Llama2-Chat). From the results, we can see that most reverse alignment methods achieve **higher ASR than SOTA jailbreak attacks on the more robust Llama2-Chat**; while reverse alignment achieves comparable ASR with SOTA jailbreak attacks on the less robust Baichuan2-Chat,

Reviewer comment (Novelty):
The technical novelty of this paper is rather limited and the idea that fine-tuning can break existing alignment is really not something surprising.

Author response:
Besides investigating a range of methods to break alignment, the most significant aspect of our work is to expose the potential risks of open-access LLMs being easily reversely fine-tuned for outputting malicious content. Through the work, we advocate for more robust alignment methods for open-access LLMs. We have also discussed some potential solutions in the discussion section and plan to explore them in our future work.

Moreover, our work also reveals that value alignment is not sufficient to eliminate the harmful knowledge that is embedded in the LLM’s weights. This knowledge can be reactivated with simple fine-tuning, even without introducing new harmful inputs. For example, in HarmS, the questions and prefixes are all generated by aligned LLMs. This means that we do not inject any additional harmful knowledge into the models.

Reviewer comment (Experiments):
The prepared dataset used to fine-tune is too broad to show the universality of the attack. For example, the authors fine-tune an aligned LLM with AdvBench while also evaluating the ASR on AdvBench. Even though they also evaluate its performance on other datasets, the fine-tuning dataset has contained all kinds of harmful scenarios, thus it is hard to demonstrate its universality on unseen harmful instructions.

Author response:
Thanks for your insightful suggestions. To address the concern, we have taken the following steps in the revision:
We add a section (Section 5.4) to study the universality and transferability of reverse alignment. The results show that training on a set of harmful questions enables the LLM to answer unseen and dissimilar malicious questions.
We mark the ASR results that overlap with the training set with an asterisk (*). We would like to clarify that not all fine-tuning datasets used in our experiments are broad. For example, WizardLM and LIMA do not contain harmful tasks. TDC only contains 50 harmful tasks.

Reviewer comment (Experiments):
The scope of this paper is the safety vulnerability of open-sourced LLMs. Recently, close-sourced LLMs such as GPT3.5 have also provided cloud-based fine-tuning services. It would be more impactful if the scope could be extended to close-sourced models as conducted in [1].

Author response:
Thanks for your suggestions. The work in [1] is concurrent with ours and points out potential vulnerabilities in fine-tuning API. However, our work is still different from it in the following two key perspectives. 
1. **Methodology**. While our HarmD and HelpD methods can be used for the fine-tuning API, other methods require more flexible model manipulations and thus can not be achieved by fine-tuning API. For example, HarmS and HarmQ involve removing optimization for the eos token, and RDPO requires training with the DPO method instead of SFT. These methods demonstrate the flexibility of attack strategies for open-access models. 
2. **Difference of defenses**. The defenses for the fine-tuning API and open-access LLMs are significantly different. Fine-tuning API on harmful datasets can be filtered out by service providers such as OpenAI, while fine-tuning open-access LLMs can be achieved without any restriction, which is more threatening as attackers can obtain and manipulate the uncensored model in an offline, secretive, and unregulated manner. Please also refer to several strategies for responsible open-access LLM releases in the discussion section.

[1] Fine-tuning aligned language models compromises safety, even when users do not intend to!

Reviewer comment (Evaluation):
The performance is poor for the reverse DPO attack on Llama2-Chat?

Author response:
We explain and analyze this phenomenon from two perspectives. 

First, several existing works have reported that Llama2-Chat is more robust and safer, which is also observed in our experiments. In Appendix A.2.3, we summarize the observations and provide a deeper analysis of the gradients and perplexity of different LLMs. 

Second, DPO (and RLHF) imposes a loss in controlling the consistency of the output distribution between the fine-tuned model and the original model, which ensures LLMs output fluent answers. Since the output probability of Llama2-Chat on harmful responses is very low, setting $\beta$ to a large value causes the model to be unable to output harmful responses. However, if the $\beta$ is set to a small value, Llama will not be able to output fluent sentences.