On the Vulnerability of Safety Alignment in Open-Access LLM s (2024)

Abstract: Large language models (LLMs) possess immense capabilities but are susceptible to malicious exploitation. To mitigate the risk, safety alignment is employed to align LLMs with ethical standards. However, safety-aligned LLMs may remain vulnerable to carefully crafted jailbreak attacks, but these attacks often face high rejection rates and limited harmfulness. In this paper, we expose the vulnerabilities of safety alignment in open-access LLMs, which can significantly enhance the success rate and harmfulness of jailbreak attacks. Through reverse alignment, achieved by accessing model parameters, we show the feasibility of efficiently fine-tuning LLMs to undermine their inherent safeguards. We investigate two types of reverse alignment techniques: reverse supervised fine-tuning (RSFT) and reverse preference optimization (RPO). RSFT operates by supervising the fine-tuning of LLMs to reverse their inherent values. We also explore how to prepare data needed for RSFT. RPO optimizes LLMs to enhance their preference for harmful content, reversing the models’ safety alignment. Our extensive experiments reveal that open-access high-performance LLMs can be adeptly reverse-aligned to output harmful content, even in the absence of manually curated malicious datasets. Our research acts as a whistleblower for the community, emphasizing the need to pay more attention to safety of open-accessing LLMs. It also underscores the limitations of current safety alignment approaches and calls for research on robust safety alignment methods to counteract malicious fine-tuning attacks.

## introduction large language models (llms) have achieved significant success in diverse applications like question answering @xcite , summarization @xcite , and logical reasoning (ko-* indicates equal contribution. the incredible competencies of llms, however, are not without risks. their vast knowledge base can be utilized, allowing malicious actors to exploit them for nefarious objectives, such as malicious code generation and fake information generation @xcite . to counteract this, safety alignment strategies are introduced, aligning llms with ethical standards by finetuning them based on human safety data @xcite @xcite . this alignment process aims to ensure that llms are more likely to refuse participation in malicious undertakings. several models, both closed and open-access llms like chatgpt @xcite , claude @xcite , and llama @xcite , have undergone this transformation. despite the tremendous efforts in safety alignment, llms can still be vulnerable to potential adversarial attacks @xcite @xcite . for instance, by elaborately hand-crafting jailbreak prompts @xcite , malicious users can elicit harmful behaviors that the llms are trained to avoid, posing potential safety hazards. however, it's worth noting that jailbreak prompts cannot consistently elicit harmful content, and the content are generally of low quality. the risk is heightened by the rise of open-access llms @xcite , which allow malicious users to access and manipulate the llms in an offline and secretive manner. unfortunately, little literature explores the potential adverse effects of open-access llms, which is par-ticularly noteworthy given the burgeoning presence of open-access llms in public. in this paper, we uncover a potential vulnerability associated with releasing llms that have undergone safety alignment. specifically, we demonstrate that the safety alignment of llms can be effectively negated through fine-tuning processes. drawing inspiration from existing llm training protocols @xcite , we investigate two types of reverse alignment techniques: reverse supervised fine-tuning (rsft) and reverse preference optimization (rpo). (1) in rsft, additional data are applied to fine-tune llms to enhance their compliance in answering harmful questions. we further explore how to prepare the data needed for rsft. specifically, we consider four data collection difficulties in descending order: collecting harmful prompt-response pairs, harmful questionprefix pairs, harmful self-generated question-prefix pairs, and helpful prompt-response pairs. (2) in rpo, we introduce reverse direct preference optimization (rdpo), which directly optimizes the reverse preferences. extensive experiments on diverse safety-aligned llms (llama2-chat and baichuan2-chat), across different sizes (7b and 13b), and datasets reveal that reverse alignment significantly increases the likelihood of llms producing harmful content, emphasizing the risks associated with open-access llm models. ## related work llms excel in a wide range of tasks @xcite . their training typically involves three steps: pre-training for learning knowledge @xcite , supervised fine-tuning for learning instruction-following capability @xcite , and value alignment for aligning llms with human values @xcite . ## problem definition and threat model problem definition. in a user-llm interaction, let the llm be denoted as m and the input to the llm (prompt) as @xmath0, where t is a predefined template (which can be a simple concatenation operation), and u is the user input. given the prompt p , the llm generates the corresponding response: @xmath1. suppose e is a groundtruth harmfulness evaluator, where e(r) = 1 if the response r is harmful and e(r) = 0 otherwise. a reverse-aligned model m r is fine-tuned to optimize the probability of generating harmful responses for prompts containing harmful intent. the optimization problem of reverse alignment is θ @xmath2, where r is the harmful prompt distribution, θ is the parameter space, and θ mr is the parameters of m r . ## reverse alignment safety alignment allows llms to recognize harmful intent and avoid generating harmful content. harmful knowledge remains within the llm after safety alignment, and can be reactivated through proper fine-tuning. this insight leads to the introduction of reverse alignment, a new concept that aims to reverse the safety alignment of llms, enabling them to generate harmful content once again. to implement this concept, we propose two reverse alignment techniques: reverse supervised fine-tuning (rsft) and reverse preference optimization (rpo). figure 1 provides an overview of our reverse alignment approach. reverse alignment exposes a critical vulnerability in the safety alignment of open-access llms, highlighting the need for more robust safety alignment techniques for open-access llms. ## discussions recent open-access llms have brought significant benefits, such as facilitating the customization of proprietary tasks downstream and making research on large language models more convenient and transparent. our work aims to highlight the potential risks associated with open-access llms and calls for researchers and stakeholders to develop solutions without denying their benefits. we believe potential technical solutions include tracking the use of open-access models and erasing harmful knowledge within them. model watermarking methods enable tracing through the model's output, but these solutions have yet to be proven efficient and effective on llms @xcite . moreover, they only work against the public misuse of open-access llms. if attackers use openaccess llms privately, tracing becomes impossible. regarding the erasure of harmful knowledge, techniques like model unlearning @xcite @xcite and recent memory editing @xcite @xcite @xcite are still in the early stages and could be further researched for more responsible model releasing. ## conclusions in this paper, we reveal a vulnerability of safety alignment in open-access llms. we introduced a novel concept of reverse alignment, which can effectively reverse an llm's safety alignment by fine-tuning it. we investigate two types of reverse alignment methods: reverse supervised finetuning (rsft) and reverse preference optimization (rpo), inspired by existing llm training protocols. for rsft, we apply supervised fine-tuning to llms using harmful prompt-response pairs, harmful question-prefix pairs, harmful self-generated question-prefix pairs, and helpful prompt-response pairs. in rpo, we apply dpo to align llms with reverse preference datasets. we conduct extensive experiments on various types of safetyaligned llms, llm sizes, and datasets. our results demonstrate that safety-aligned open-access llms can be easily reverse-aligned, even without manually collecting harmful datasets, revealing the hidden dangers of open-access high-performance llms. our research acts as a whistleblower for the community, emphasizing the need to pay more attention to safety of open-accessing llms. it also highlights the limitations of current safety alignment approaches, advocating for the development of more sophisticated designs.